-- Logs begin at Fri 2022-10-28 19:16:05 UTC, end at Fri 2022-10-28 20:54:31 UTC. -- Oct 28 19:45:50.736899 dv-cinder-862365-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 28 19:45:50.744888 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Oct 28 19:45:50 2022] *** Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: nodename: dv-cinder-862365-2-iscsi Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: machine: x86_64 Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: clock source: unix Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: pcre jit disabled Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: detected number of CPU cores: 4 Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: current working directory: / Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: detected binary path: /usr/bin/uwsgi-core Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: your processes number limit is 47680 Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: your memory page size is 4096 bytes Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: detected max file descriptor number: 2048 Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: lock engine: pthread robust mutexes Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: thunder lock: enabled Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 28 19:45:50.751095 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 28 19:45:50.787763 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: Python main interpreter initialized at 0x56526af929c0 Oct 28 19:45:50.787763 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: python threads support enabled Oct 28 19:45:50.787763 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: your server socket listen backlog is limited to 100 connections Oct 28 19:45:50.787763 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: your mercy for graceful operations on workers is 90 seconds Oct 28 19:45:50.788125 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: mapped 403077 bytes (393 KB) for 2 cores Oct 28 19:45:50.788199 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: *** Operational MODE: preforking *** Oct 28 19:45:50.788464 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: *** uWSGI is running in multiple interpreter mode *** Oct 28 19:45:50.788464 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: spawned uWSGI master process (pid: 81599) Oct 28 19:45:50.788669 dv-cinder-862365-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 28 19:45:50.791077 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: spawned uWSGI worker 1 (pid: 81613, cores: 1) Oct 28 19:45:50.791077 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: spawned uWSGI worker 2 (pid: 81614, cores: 1) Oct 28 19:45:50.792064 dv-cinder-862365-2-iscsi devstack@keystone.service[81599]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 28 19:45:55.188313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 19:45:55.188313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: from cryptography.utils import int_from_bytes Oct 28 19:45:55.189092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 19:45:55.189092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: from cryptography.utils import int_from_bytes Oct 28 19:45:55.191626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 19:45:55.191626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: from cryptography.utils import int_from_bytes Oct 28 19:45:55.192079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 19:45:55.192079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: from cryptography.utils import int_from_bytes Oct 28 19:45:55.243040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 28 19:45:55.243379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 28 19:45:55.243475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 28 19:45:55.243574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 28 19:45:55.243657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] command line args: [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 28 19:45:55.243736 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] command line args: [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 28 19:45:55.243815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 28 19:45:55.243892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 28 19:45:55.243974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ================================================================================ {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 28 19:45:55.244052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ================================================================================ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 28 19:45:55.244130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] admin_token = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] admin_token = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] config_dir = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config_dir = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] config_source = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] debug = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.244942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config_source = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] debug = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.245929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] list_limit = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] log_config_append = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_config_append = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246792 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.246892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] log_dir = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_dir = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] log_file = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.247946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.248931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249438 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.249897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.250059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.250167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.250238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.250449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.250575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.250938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.66/identity {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.66/identity {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] publish_errors = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] publish_errors = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.251981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] standard_threads = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] standard_threads = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.252948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_journal = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_json = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] use_journal = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.253994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_stderr = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.254117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] use_json = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.254239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_syslog = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.254356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] use_stderr = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.254470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.254596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] use_syslog = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.254713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.254828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 19:45:55.254942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.255055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.255055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.255348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.255348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.255670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.255790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.255908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.256946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.257993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.258234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.258357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.259999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.260959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.261078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.261193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.261308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.261480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.261598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.261796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.261915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.262899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.263965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.264989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.265980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.266969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.267893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.268954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269066 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.269937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.270901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.271891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.272953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.273985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.274156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.274832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.274975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.275968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.276900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.277984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.278923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.279927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.280878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.external = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.281950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] auth.external = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.password = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.token = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.282791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] auth.password = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] auth.token = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.283972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.284922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] credential.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.285919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.286887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.287959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.288160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.288282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.288399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.288516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.288633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.288750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.288882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.289003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.289120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.289302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.289476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.289594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.289760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.289887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.290955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.291818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.292936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.293904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.294990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.295933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.296917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.297895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.298958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.299904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.300979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.301978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.302203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.302319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.302435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.302561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.302688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.302805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.302922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.303982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.304907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.305003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.305099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.305196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.305300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.305494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.305593 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309177 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.309949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.310942 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.311948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.312964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.313943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314399 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314624 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.314976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.315935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] role.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] role.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.316965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.317935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318310 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.318882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.319850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.320961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.321670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.328724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.328850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.328928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.328928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.329938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.330968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331354 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.331939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332469 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.connection = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.332985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.333957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 28 19:45:55.334705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 19:45:55.334705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81614) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 28 19:45:55.360889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.361111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.361356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.363363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.363363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.363363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.369846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.370602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.398327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.398554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.399969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.400219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.458108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.458671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.516427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.516665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.516831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.517077 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.676978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.677119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.677200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.677359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.702769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.702999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703093 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.703995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.704119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.704199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.704338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.715996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.716667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.736248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.736488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.736715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.736945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.737904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.738121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.738339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81614) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 19:45:55.754688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.754905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 19:45:55.755254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.755456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 19:45:55.755735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.756002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.756269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.756533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.758011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.758229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 19:45:55.759000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.759295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 19:45:55.759585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.759859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.760131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.760400 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.766398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.766686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.766966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.767257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.770040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.770325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.770604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.770894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.774800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.777782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.778073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.778365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.778658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.778822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.778949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.779217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.779493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.779773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.780051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.780360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.780635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.781756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.782052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.782345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.782640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.782934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.783205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.783486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.783771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.784052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.784336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.784612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.803923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.805436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.805729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.809008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.810505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.810757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.810841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.811049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.811339 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.815431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.815731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.816023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.817701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.817995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.822849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.823149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.823742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.824051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.826277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.826277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.826277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.826277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.826277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.826277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.828978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.829278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.829601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.829895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.830180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.830464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.830749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.831025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.841835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.842122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.846003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.847556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.847851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.849219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.849550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.851683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.854406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.856084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.856401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.857601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.860708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.860800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.861144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.861467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.861762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.862054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.862347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.863904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.867137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.867445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.867740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.873577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.873577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.873577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.878237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.878580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.878861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.879130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.885818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.886116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.886391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.886661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.886744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.888212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.889847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.890129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.892831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.893122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.893440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.893523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.893731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.898444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.898522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.898522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.898522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.900569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.900868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.901160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.901485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.901875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.906020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.906311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.906624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.906908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.907306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.907593 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.907878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.912925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.913219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.913538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.913826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.914226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.914513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.915267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.923252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.931121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.931121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.931121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.931372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.932127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.932417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.932829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.934154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.937115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.938580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.939472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.940079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.940372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.940777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.940957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.943892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.944295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.944591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.944880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.948133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.949649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.952525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.952930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.953217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.953302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.953532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.953614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.953882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.960873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.961167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.961369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.961482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.961842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.962152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.962248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.962397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.962482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.962676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.962965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.963242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.963570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.963848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.964116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.964441 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.964714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.969301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81614) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 19:45:55.969820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.969916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.970177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.970461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.970744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.971028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.971320 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.971600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.971886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.972164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.972485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.972768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.973045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81614) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 19:45:55.999934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x56526af929c0 pid: 81613 (default app) Oct 28 19:45:56.003324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c83a25-8b2e-41cd-b213-167504f6148a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:45:56.003621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c83a25-8b2e-41cd-b213-167504f6148a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:45:56.003900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c83a25-8b2e-41cd-b213-167504f6148a None None] PATH_INFO: `/v3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:45:56.008108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1/1] 10.222.0.66 () {50 vars in 791 bytes} [Fri Oct 28 19:45:56 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:45:56.008291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x56526af929c0 pid: 81614 (default app) Oct 28 19:46:03.788502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577c2341-9606-48d2-b16a-54803ba918b5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:03.789035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577c2341-9606-48d2-b16a-54803ba918b5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:03.789035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577c2341-9606-48d2-b16a-54803ba918b5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:03.790552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 2/2] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:03.799762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:03.799762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:03.799762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:03.869730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: INFO dbcounter [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] Registered counter for database keystone Oct 28 19:46:03.889036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG oslo_db.sqlalchemy.engines [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81614) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 28 19:46:03.895209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writer thread running {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 28 19:46:04.074700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG passlib.handlers.bcrypt [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81614) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 28 19:46:04.075022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG passlib.handlers.bcrypt [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81614) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 28 19:46:04.132376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:04.134672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:04.262272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1/3] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:03 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 467 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 28 19:46:04.754224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577c2341-9606-48d2-b16a-54803ba918b5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:04.823150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: INFO dbcounter [None req-577c2341-9606-48d2-b16a-54803ba918b5 None None] Registered counter for database keystone Oct 28 19:46:04.844879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG oslo_db.sqlalchemy.engines [None req-577c2341-9606-48d2-b16a-54803ba918b5 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81613) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 28 19:46:04.851253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writer thread running {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 28 19:46:04.969335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577c2341-9606-48d2-b16a-54803ba918b5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:04.973551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b45b35-4254-40b7-beab-8396ec538fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:04.974170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b45b35-4254-40b7-beab-8396ec538fd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:04.974451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b45b35-4254-40b7-beab-8396ec538fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:04.974731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b45b35-4254-40b7-beab-8396ec538fd6 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:04.982957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b45b35-4254-40b7-beab-8396ec538fd6 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:05.035527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING keystone.common.rbac_enforcer.enforcer [None req-25b45b35-4254-40b7-beab-8396ec538fd6 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 28 19:46:05.035876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b45b35-4254-40b7-beab-8396ec538fd6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:05.045710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 3/4] 10.222.0.66 () {58 vars in 1324 bytes} [Fri Oct 28 19:46:04 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 19:46:05.051569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:05.066357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc20880e-d853-451d-8fa4-4bb6abada692 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:05.071306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce12352b-fb84-4ecf-b2c4-5db95849bdc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:05.072397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce12352b-fb84-4ecf-b2c4-5db95849bdc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:05.072680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce12352b-fb84-4ecf-b2c4-5db95849bdc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:05.072953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce12352b-fb84-4ecf-b2c4-5db95849bdc3 admin admin] PATH_INFO: `/v3/projects/9255f3850e094deb824d58d03e377387` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:05.076192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce12352b-fb84-4ecf-b2c4-5db95849bdc3 admin admin] RBAC: Authorizing `identity:get_project(project_id=9255f3850e094deb824d58d03e377387)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:05.128968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING keystone.common.rbac_enforcer.enforcer [None req-ce12352b-fb84-4ecf-b2c4-5db95849bdc3 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 28 19:46:05.129303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce12352b-fb84-4ecf-b2c4-5db95849bdc3 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:05.134443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 2/5] 10.222.0.66 () {56 vars in 1207 bytes} [Fri Oct 28 19:46:05 2022] GET /identity/v3/projects/9255f3850e094deb824d58d03e377387 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:07.249274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e42b35-dcf7-4964-970e-b2ba1d5d7c8a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:07.249811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e42b35-dcf7-4964-970e-b2ba1d5d7c8a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:07.249892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e42b35-dcf7-4964-970e-b2ba1d5d7c8a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:07.253903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 4/6] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:07 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:07.259572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b380b6-77a7-4675-915d-a8fea47500b4 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:07.260039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b380b6-77a7-4675-915d-a8fea47500b4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:07.260126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b380b6-77a7-4675-915d-a8fea47500b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:07.280176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-07b380b6-77a7-4675-915d-a8fea47500b4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:07.281653 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-07b380b6-77a7-4675-915d-a8fea47500b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:07.290023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 3/7] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:07 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 28 19:46:07.559625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e42b35-dcf7-4964-970e-b2ba1d5d7c8a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:07.572049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e42b35-dcf7-4964-970e-b2ba1d5d7c8a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:07.576260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b995f7-9c0e-482b-a478-7a01a8fec8d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:07.576857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b995f7-9c0e-482b-a478-7a01a8fec8d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:07.577141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b995f7-9c0e-482b-a478-7a01a8fec8d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:07.577446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b995f7-9c0e-482b-a478-7a01a8fec8d5 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:07.584969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b995f7-9c0e-482b-a478-7a01a8fec8d5 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:07.586885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b995f7-9c0e-482b-a478-7a01a8fec8d5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:07.595418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 5/8] 10.222.0.66 () {58 vars in 1324 bytes} [Fri Oct 28 19:46:07 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 19:46:07.600896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b380b6-77a7-4675-915d-a8fea47500b4 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:07.610146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b380b6-77a7-4675-915d-a8fea47500b4 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:07.614499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4166de0b-2f88-4a7d-b77c-eed37b647356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:07.615127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166de0b-2f88-4a7d-b77c-eed37b647356 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:07.615412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166de0b-2f88-4a7d-b77c-eed37b647356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:07.615691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166de0b-2f88-4a7d-b77c-eed37b647356 admin admin] PATH_INFO: `/v3/users/8481404e88e44e68b2c70669768eebe2` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:07.617018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4166de0b-2f88-4a7d-b77c-eed37b647356 admin admin] RBAC: Authorizing `identity:get_user(user_id=8481404e88e44e68b2c70669768eebe2)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:07.619489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4166de0b-2f88-4a7d-b77c-eed37b647356 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:07.621000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 4/9] 10.222.0.66 () {56 vars in 1198 bytes} [Fri Oct 28 19:46:07 2022] GET /identity/v3/users/8481404e88e44e68b2c70669768eebe2 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:09.802067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640f0562-2877-4fc1-a006-ed25cc4b4885 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:09.802691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640f0562-2877-4fc1-a006-ed25cc4b4885 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:09.802691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640f0562-2877-4fc1-a006-ed25cc4b4885 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:09.803924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 6/10] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:09.809928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b3ca8-957c-43a5-939d-31fd978a48ea None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:09.810342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b3ca8-957c-43a5-939d-31fd978a48ea None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:09.810500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b3ca8-957c-43a5-939d-31fd978a48ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:09.828956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0e1b3ca8-957c-43a5-939d-31fd978a48ea None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:09.830179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0e1b3ca8-957c-43a5-939d-31fd978a48ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:09.889691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890c4e3-5e2b-4706-9df1-2c11c74af1af None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:09.889977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890c4e3-5e2b-4706-9df1-2c11c74af1af None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:09.890259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890c4e3-5e2b-4706-9df1-2c11c74af1af None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:09.891580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 7/11] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:09.896550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:09.896840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:09.897387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:09.909513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 5/12] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:09.941005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG passlib.handlers.bcrypt [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81613) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 28 19:46:09.941687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG passlib.handlers.bcrypt [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81613) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 28 19:46:09.990909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:09.993406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:10.109778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 8/13] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:10.184188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b3ca8-957c-43a5-939d-31fd978a48ea None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.208071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:10.286292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b3ca8-957c-43a5-939d-31fd978a48ea None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.319915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.321049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.321354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.321641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.332505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:10.342129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:10.343056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.344129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 6/14] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:46:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:10.349127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.366603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.450172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:10.450614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0f25e9-370a-48da-a10b-1aec3de10258 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.452941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:10.468579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa4d11-fa65-47b2-bc72-3259b53bce49 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.488008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.488887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.489176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.489481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.490377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:10.490684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:10.492121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:10.503927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.504545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.504830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.505101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.505371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 9/15] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:10.505800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:10.507204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:10.511663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.542121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.sql.core [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 28 19:46:10.542121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 19:46:10.542121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [parameters: {'id': 'ef7d7a39d744440db9a2d674983fbff3', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 28 19:46:10.542121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81614) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:46:10.560197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 28 19:46:10.561081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application yield resource Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise self.value Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application session.commit() Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 19:46:10.562190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.flush() Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application compat.raise_( Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 19:46:10.563693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 19:46:10.571801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:46:10.572899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [parameters: {'id': 'ef7d7a39d744440db9a2d674983fbff3', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 28 19:46:10.574146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:10.575077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 7/16] 10.222.0.66 () {60 vars in 1137 bytes} [Fri Oct 28 19:46:10 2022] POST /identity/v3/domains => generated 190 bytes in 195 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:10.576119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.617603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c0bbd-ffe4-4373-b61e-13084aeb3ed3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.650946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.651801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.652086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.652356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] PATH_INFO: `/v3/users/8481404e88e44e68b2c70669768eebe2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.653878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] RBAC: Authorizing `identity:get_user(user_id=8481404e88e44e68b2c70669768eebe2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:10.655330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:10.657051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 10/17] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:10 2022] GET /identity/v3/users/8481404e88e44e68b2c70669768eebe2 => generated 247 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:10.662401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.668033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf28a6d-e102-4841-af65-123ed428ae64 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.695780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:10.702361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.702999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.703286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.703568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.714575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:10.716056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:10.722761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:46:10.731493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:10.732649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:46:10.733586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:46:10.733586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:46:10.733586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:10.733586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 8/18] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:10.738527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.761632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0178f802-1b8b-4fd8-97e1-f84f133ee342 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.796650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.797287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.797605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.797889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.799022 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:10.800490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:10.802078 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 11/19] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:10.807751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:10.842045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908aba8d-4c0c-4678-8d28-86c5d7bb1361 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.876206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77674d6-e8f2-4149-90d6-149d8c2eaf6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.876757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77674d6-e8f2-4149-90d6-149d8c2eaf6d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.877044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77674d6-e8f2-4149-90d6-149d8c2eaf6d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.877352 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77674d6-e8f2-4149-90d6-149d8c2eaf6d None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.877997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77674d6-e8f2-4149-90d6-149d8c2eaf6d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:10.878344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77674d6-e8f2-4149-90d6-149d8c2eaf6d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:10.879735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77674d6-e8f2-4149-90d6-149d8c2eaf6d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:10.897522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 9/20] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:46:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:10.906434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f3d8fb-dba2-4b2d-9426-b7bd64da884f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:10.908210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:10.945118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae26b4-34dd-409a-976e-58187bc4c176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:10.946380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae26b4-34dd-409a-976e-58187bc4c176 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:10.946662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae26b4-34dd-409a-976e-58187bc4c176 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:10.946939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae26b4-34dd-409a-976e-58187bc4c176 None admin] PATH_INFO: `/v3/domains/default/users/8481404e88e44e68b2c70669768eebe2/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:10.949026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ae26b4-34dd-409a-976e-58187bc4c176 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=8481404e88e44e68b2c70669768eebe2, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:10.950580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ae26b4-34dd-409a-976e-58187bc4c176 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:11.132167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 12/21] 10.222.0.66 () {58 vars in 1361 bytes} [Fri Oct 28 19:46:10 2022] PUT /identity/v3/domains/default/users/8481404e88e44e68b2c70669768eebe2/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:46:13.432546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313bf871-c0b3-4739-930f-e61f38f6e645 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:13.433046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313bf871-c0b3-4739-930f-e61f38f6e645 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:13.433046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313bf871-c0b3-4739-930f-e61f38f6e645 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:13.435026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 10/22] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:13.440997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea135a8-5842-454b-8c2d-bd20fed55707 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:13.441405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea135a8-5842-454b-8c2d-bd20fed55707 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:13.441586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea135a8-5842-454b-8c2d-bd20fed55707 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:13.470443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-aea135a8-5842-454b-8c2d-bd20fed55707 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:13.472008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aea135a8-5842-454b-8c2d-bd20fed55707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:13.559911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 13/23] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:14.061481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836eefb4-23d9-4de2-b79e-e90697e5f4be None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.062144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836eefb4-23d9-4de2-b79e-e90697e5f4be None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.062613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836eefb4-23d9-4de2-b79e-e90697e5f4be None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.064145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 11/24] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:14.069894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c43f837-67df-45a2-a900-98a51fdb0bb8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.070196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c43f837-67df-45a2-a900-98a51fdb0bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.070519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c43f837-67df-45a2-a900-98a51fdb0bb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.095255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836eefb4-23d9-4de2-b79e-e90697e5f4be None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:14.096889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0c43f837-67df-45a2-a900-98a51fdb0bb8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:14.098350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0c43f837-67df-45a2-a900-98a51fdb0bb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:14.202480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 14/25] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:14.220841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:14.240511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c0dab-32e7-4606-a89b-535a3a2ab429 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.241146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c0dab-32e7-4606-a89b-535a3a2ab429 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.241619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c0dab-32e7-4606-a89b-535a3a2ab429 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.251299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 15/26] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:14 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:14.257091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed4467-c9cb-43d6-a895-21e9144f37d9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.257793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed4467-c9cb-43d6-a895-21e9144f37d9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.258526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed4467-c9cb-43d6-a895-21e9144f37d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.301248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-70ed4467-c9cb-43d6-a895-21e9144f37d9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:14.302431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-70ed4467-c9cb-43d6-a895-21e9144f37d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:14.303477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836eefb4-23d9-4de2-b79e-e90697e5f4be None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:14.347362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:14.369434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:14.370062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.370346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.370621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.371315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:14.372717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:14.444329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 16/27] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:14.445793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028439da-3931-4afd-877a-23be5edc1074 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.446085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028439da-3931-4afd-877a-23be5edc1074 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.446369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028439da-3931-4afd-877a-23be5edc1074 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.448045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 17/28] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:14.454322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d6358-bd08-45ae-82a3-b88d44bf731d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.454613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d6358-bd08-45ae-82a3-b88d44bf731d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.454898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d6358-bd08-45ae-82a3-b88d44bf731d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.476783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0c0d6358-bd08-45ae-82a3-b88d44bf731d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:14.478097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0c0d6358-bd08-45ae-82a3-b88d44bf731d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:14.603355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 18/29] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:14.641332 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 12/30] 10.222.0.66 () {60 vars in 1131 bytes} [Fri Oct 28 19:46:14 2022] POST /identity/v3/roles => generated 219 bytes in 546 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:46:14.659918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d6358-bd08-45ae-82a3-b88d44bf731d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:14.759332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:14.783647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d6358-bd08-45ae-82a3-b88d44bf731d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:14.833836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04aaf525-0eb0-40bd-a499-ae7ee96f77f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:14.834885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04aaf525-0eb0-40bd-a499-ae7ee96f77f6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.834989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:14.835501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04aaf525-0eb0-40bd-a499-ae7ee96f77f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.836016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04aaf525-0eb0-40bd-a499-ae7ee96f77f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.866506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04aaf525-0eb0-40bd-a499-ae7ee96f77f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:14.868610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04aaf525-0eb0-40bd-a499-ae7ee96f77f6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-04aaf525-0eb0-40bd-a499-ae7ee96f77f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:46:14.890304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:14.891232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:46:14.892032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:46:14.892032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:46:14.892032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:14.897943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 19/31] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:14.900089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d27fe9c-a702-4ae6-8766-09297dd44305 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:14.900573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d27fe9c-a702-4ae6-8766-09297dd44305 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:14.901060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d27fe9c-a702-4ae6-8766-09297dd44305 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:14.909059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 20/32] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:14 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:14.910034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d27fe9c-a702-4ae6-8766-09297dd44305 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:14.969708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d5ca33-b691-48c6-8a19-366f46a93363 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:14.994287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:15.015379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2022e0-d9b0-4ba8-b4b8-521b2926dfb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.016169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2022e0-d9b0-4ba8-b4b8-521b2926dfb7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.016457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2022e0-d9b0-4ba8-b4b8-521b2926dfb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.016734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2022e0-d9b0-4ba8-b4b8-521b2926dfb7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.033281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2022e0-d9b0-4ba8-b4b8-521b2926dfb7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-0c2022e0-d9b0-4ba8-b4b8-521b2926dfb7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:15.040369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:15.041513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:15.042283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:15.042846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 13/33] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:46:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:15.043889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5378405d-3ff0-4f16-8753-c6421abe37d9 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.044184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5378405d-3ff0-4f16-8753-c6421abe37d9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.044464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5378405d-3ff0-4f16-8753-c6421abe37d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.069532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5378405d-3ff0-4f16-8753-c6421abe37d9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:15.071899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5378405d-3ff0-4f16-8753-c6421abe37d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:15.083535 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d27fe9c-a702-4ae6-8766-09297dd44305 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:15.124289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.125294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.125610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.126031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.126824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:15.127115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:15.129144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:15.131044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:15.145903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 14/34] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:15.147075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5378405d-3ff0-4f16-8753-c6421abe37d9 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.149100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 21/35] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:46:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:15.153793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.275820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3deaf-6219-4c0b-8244-bfca625c0f2c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:15.277314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5378405d-3ff0-4f16-8753-c6421abe37d9 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:15.289935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:15.316538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea95cf0-5f1c-4a17-aa5b-17e9ad02a4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.317170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea95cf0-5f1c-4a17-aa5b-17e9ad02a4d1 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.317544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea95cf0-5f1c-4a17-aa5b-17e9ad02a4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.317841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea95cf0-5f1c-4a17-aa5b-17e9ad02a4d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.318572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea95cf0-5f1c-4a17-aa5b-17e9ad02a4d1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:15.318737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.319354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.319978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea95cf0-5f1c-4a17-aa5b-17e9ad02a4d1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:15.320079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.320301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.321727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:15.322512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:15.324041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:15.335720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 15/36] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:15.337222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.389732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 22/37] 10.222.0.66 () {60 vars in 1131 bytes} [Fri Oct 28 19:46:15 2022] POST /identity/v3/roles => generated 225 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:15.390908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c306a1-59dc-4808-998a-0b608fc142d0 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.391245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c306a1-59dc-4808-998a-0b608fc142d0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.391531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c306a1-59dc-4808-998a-0b608fc142d0 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.392974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 23/38] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:15.397929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c306a1-59dc-4808-998a-0b608fc142d0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.462770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8426013-50f9-413c-97e1-bf75f5cb062e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:15.469380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:15.499351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.499993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.500280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.500558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.501284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:15.502835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:15.528211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c306a1-59dc-4808-998a-0b608fc142d0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:15.566365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6e09d-b654-464a-a645-c1d5e4b5ed46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.567041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6e09d-b654-464a-a645-c1d5e4b5ed46 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.567325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6e09d-b654-464a-a645-c1d5e4b5ed46 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.567603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6e09d-b654-464a-a645-c1d5e4b5ed46 None admin] PATH_INFO: `/v3/users/8481404e88e44e68b2c70669768eebe2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.568732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c6e09d-b654-464a-a645-c1d5e4b5ed46 None admin] RBAC: Authorizing `identity:get_user(user_id=8481404e88e44e68b2c70669768eebe2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:15.570235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c6e09d-b654-464a-a645-c1d5e4b5ed46 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:15.572266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 24/39] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:15 2022] GET /identity/v3/users/8481404e88e44e68b2c70669768eebe2 => generated 247 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:15.573557 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a81e777-c5fb-42dd-b195-991af4f27788 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.573849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a81e777-c5fb-42dd-b195-991af4f27788 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.574128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a81e777-c5fb-42dd-b195-991af4f27788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.593999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8a81e777-c5fb-42dd-b195-991af4f27788 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:15.598007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8a81e777-c5fb-42dd-b195-991af4f27788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:15.625723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 16/40] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:46:15 2022] POST /identity/v3/projects => generated 301 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:15.626377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.663235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:15.677350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 25/41] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:15.677350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a81e777-c5fb-42dd-b195-991af4f27788 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.748001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:15.772880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1aced2-dba1-4c66-9676-c76d27653037 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:15.815713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a81e777-c5fb-42dd-b195-991af4f27788 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:15.821972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.822722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.823117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.824030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.824833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:15.827001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:15.855831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:15.856695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:15.857090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:15.857516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:15.858749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:15.860236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:15.861933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 26/42] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:15.867880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.892043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 17/43] 10.222.0.66 () {60 vars in 1131 bytes} [Fri Oct 28 19:46:15 2022] POST /identity/v3/roles => generated 223 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:15.953155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:15.982888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b0d67-f1cc-4801-8517-07db18269657 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:16.006995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:16.028317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:16.028317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:16.028317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:16.028317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:16.028738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.domain.id=default, user.id=8481404e88e44e68b2c70669768eebe2) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:16.028738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:16.029761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:16.039203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 27/44] 10.222.0.66 () {56 vars in 1322 bytes} [Fri Oct 28 19:46:15 2022] GET /identity/v3/role_assignments?user.id=8481404e88e44e68b2c70669768eebe2&scope.domain.id=default&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 530 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:16.058644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:16.080886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ed094b-d4fc-4e30-bb92-de5b3d7340bb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:16.122411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f971c-1015-4a5c-8979-9536535d76c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:16.123353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f971c-1015-4a5c-8979-9536535d76c3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:16.123775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f971c-1015-4a5c-8979-9536535d76c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:16.124226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f971c-1015-4a5c-8979-9536535d76c3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:16.125911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7f971c-1015-4a5c-8979-9536535d76c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:16.128642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7f971c-1015-4a5c-8979-9536535d76c3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:16.130401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 18/45] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:16.136450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:16.243540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b42559-b6c4-4296-8de4-847b8b98cf59 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:16.277132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3bc5ee-d319-46c1-8b71-bb54a8139143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:16.277769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bc5ee-d319-46c1-8b71-bb54a8139143 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:16.278054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bc5ee-d319-46c1-8b71-bb54a8139143 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:16.278332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bc5ee-d319-46c1-8b71-bb54a8139143 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:16.278990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3bc5ee-d319-46c1-8b71-bb54a8139143 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:16.280454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3bc5ee-d319-46c1-8b71-bb54a8139143 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:16.365585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 28/46] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:46:16 2022] POST /identity/v3/projects => generated 312 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:18.509504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53df535c-9907-4468-ad68-116fe938ed2d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:18.510040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53df535c-9907-4468-ad68-116fe938ed2d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:18.510127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53df535c-9907-4468-ad68-116fe938ed2d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:18.511452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 19/47] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:18.516618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec470f5-7366-4592-99f3-821dd39fcff1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:18.517020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec470f5-7366-4592-99f3-821dd39fcff1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:18.517186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec470f5-7366-4592-99f3-821dd39fcff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:18.534970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fec470f5-7366-4592-99f3-821dd39fcff1 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:18.536148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fec470f5-7366-4592-99f3-821dd39fcff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:18.564410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:18.602384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 29/48] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:18.875914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53df535c-9907-4468-ad68-116fe938ed2d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:18.975645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53df535c-9907-4468-ad68-116fe938ed2d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:19.115158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73922f94-13ba-4856-8dd7-5cd98a33660d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:19.115790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73922f94-13ba-4856-8dd7-5cd98a33660d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:19.116078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73922f94-13ba-4856-8dd7-5cd98a33660d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:19.116358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73922f94-13ba-4856-8dd7-5cd98a33660d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:19.118366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73922f94-13ba-4856-8dd7-5cd98a33660d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:19.119819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73922f94-13ba-4856-8dd7-5cd98a33660d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:19.121393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 20/49] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:19.127272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec470f5-7366-4592-99f3-821dd39fcff1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:19.223779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec470f5-7366-4592-99f3-821dd39fcff1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:19.257237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a40163-287d-408b-ae29-7a2401fa412b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:19.257891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a40163-287d-408b-ae29-7a2401fa412b None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:19.258176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a40163-287d-408b-ae29-7a2401fa412b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:19.258455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a40163-287d-408b-ae29-7a2401fa412b None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:19.259106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a40163-287d-408b-ae29-7a2401fa412b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:19.260556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a40163-287d-408b-ae29-7a2401fa412b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:19.331352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 30/50] 10.222.0.66 () {60 vars in 1140 bytes} [Fri Oct 28 19:46:19 2022] POST /identity/v3/projects => generated 298 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:21.504643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143b08cb-21ed-49fa-bfc0-400ddd20ef84 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:21.505155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143b08cb-21ed-49fa-bfc0-400ddd20ef84 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:21.505237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143b08cb-21ed-49fa-bfc0-400ddd20ef84 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:21.506583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 21/51] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:21.511963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6d8bc-88b8-4b03-b73f-e44e6499921c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:21.512342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6d8bc-88b8-4b03-b73f-e44e6499921c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:21.512515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6d8bc-88b8-4b03-b73f-e44e6499921c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:21.633795 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-96b6d8bc-88b8-4b03-b73f-e44e6499921c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:21.635003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-96b6d8bc-88b8-4b03-b73f-e44e6499921c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:21.648112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:21.699961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 31/52] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:21.970493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143b08cb-21ed-49fa-bfc0-400ddd20ef84 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:21.989626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:22.072210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143b08cb-21ed-49fa-bfc0-400ddd20ef84 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:22.105354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83096079-82c9-42c1-812e-b0f3d4362b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:22.105994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83096079-82c9-42c1-812e-b0f3d4362b57 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:22.106281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83096079-82c9-42c1-812e-b0f3d4362b57 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:22.106561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83096079-82c9-42c1-812e-b0f3d4362b57 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:22.107671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83096079-82c9-42c1-812e-b0f3d4362b57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:22.109122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83096079-82c9-42c1-812e-b0f3d4362b57 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:22.110728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 22/53] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:22.116412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b6d8bc-88b8-4b03-b73f-e44e6499921c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:22.210378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b6d8bc-88b8-4b03-b73f-e44e6499921c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:22.244049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf5d7ed-a7cd-4650-b5cd-b0713b1846d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:22.244582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf5d7ed-a7cd-4650-b5cd-b0713b1846d2 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:22.244870 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf5d7ed-a7cd-4650-b5cd-b0713b1846d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:22.245146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf5d7ed-a7cd-4650-b5cd-b0713b1846d2 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:22.245831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf5d7ed-a7cd-4650-b5cd-b0713b1846d2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:22.247264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf5d7ed-a7cd-4650-b5cd-b0713b1846d2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:22.364604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 32/54] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:46:22 2022] POST /identity/v3/users => generated 275 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:46:25.961368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b003a78-85f2-4704-93be-2ec7997524cc None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:25.961368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b003a78-85f2-4704-93be-2ec7997524cc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:25.961368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b003a78-85f2-4704-93be-2ec7997524cc None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:25.963018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 23/55] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:25.968635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e83e397-b5a9-43bd-ade0-d278c4f6a2b5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:25.969572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e83e397-b5a9-43bd-ade0-d278c4f6a2b5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:25.970093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e83e397-b5a9-43bd-ade0-d278c4f6a2b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:25.981870 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:25.997194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5e83e397-b5a9-43bd-ade0-d278c4f6a2b5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:25.998509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5e83e397-b5a9-43bd-ade0-d278c4f6a2b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:26.014344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4f257f-5157-4ec2-9800-345868e7fae4 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.014634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4f257f-5157-4ec2-9800-345868e7fae4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.017031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4f257f-5157-4ec2-9800-345868e7fae4 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.017031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 24/56] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:26.022615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73355b1e-ab8a-4cfe-96c4-d16690ff89c5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.023077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73355b1e-ab8a-4cfe-96c4-d16690ff89c5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.023935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73355b1e-ab8a-4cfe-96c4-d16690ff89c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.053833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-73355b1e-ab8a-4cfe-96c4-d16690ff89c5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:26.056903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-73355b1e-ab8a-4cfe-96c4-d16690ff89c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:26.129069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 33/57] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:26.160337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 25/58] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:26.178007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bccb724-616b-4063-83f5-68141aaf426a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.178319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bccb724-616b-4063-83f5-68141aaf426a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.178764 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bccb724-616b-4063-83f5-68141aaf426a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.181168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 34/59] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:26.186337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51e180c-ff85-44bf-9ab8-13820b3e27b7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.186880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51e180c-ff85-44bf-9ab8-13820b3e27b7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.187386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51e180c-ff85-44bf-9ab8-13820b3e27b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.217876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e51e180c-ff85-44bf-9ab8-13820b3e27b7 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:26.219376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e51e180c-ff85-44bf-9ab8-13820b3e27b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:26.241734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:26.322776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 26/60] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:26.369573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e24072-013b-4fb1-b74d-7fcafcc005a6 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.370215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e24072-013b-4fb1-b74d-7fcafcc005a6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.370721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e24072-013b-4fb1-b74d-7fcafcc005a6 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.372286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 35/61] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:26.377957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830a377-bad6-471b-a3b5-4713b5930db2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.378251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830a377-bad6-471b-a3b5-4713b5930db2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.378537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830a377-bad6-471b-a3b5-4713b5930db2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.401709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8830a377-bad6-471b-a3b5-4713b5930db2 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:26.402885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8830a377-bad6-471b-a3b5-4713b5930db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:26.410831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4841c45-a90f-4c5c-80ea-aa787ee0647e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.411369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4841c45-a90f-4c5c-80ea-aa787ee0647e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.411865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4841c45-a90f-4c5c-80ea-aa787ee0647e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.414554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 36/62] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:26.418528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a082a8-c27f-4de0-a52e-d4d8338effde None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.419072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a082a8-c27f-4de0-a52e-d4d8338effde None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.419570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a082a8-c27f-4de0-a52e-d4d8338effde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.454388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c5a082a8-c27f-4de0-a52e-d4d8338effde None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:26.455580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c5a082a8-c27f-4de0-a52e-d4d8338effde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:26.495936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 27/63] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:26.496862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8830a377-bad6-471b-a3b5-4713b5930db2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:26.553702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 37/64] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:26.562611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a082a8-c27f-4de0-a52e-d4d8338effde None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:26.584879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:26.631840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8830a377-bad6-471b-a3b5-4713b5930db2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:26.666604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:26.678995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:26.679685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.679972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.680255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] PATH_INFO: `/v3/roles/member` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.695655 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:26.702162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:26.703027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:26.703773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:26.706609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 28/65] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:26.708858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:26.721082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a082a8-c27f-4de0-a52e-d4d8338effde None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:26.766853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:26.767538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.767889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.768236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.782769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:26.789976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:26.790925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:26.791696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 38/66] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:46:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:26.796602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:26.824822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c77d881-e470-4c80-93ac-70d244144ebe None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:26.862333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:26.863054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.863331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.863607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.866258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:26.875073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:26.880130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:26.881160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:26.882156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 29/67] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:46:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:26.882946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:26.903510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cc0e7f-a7f2-43b5-97e8-e3b70969f5ae None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:26.927206 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:26.938778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:26.939401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:26.939890 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:26.939997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:26.940684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:26.940999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:26.942431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:26.950899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 39/68] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:26.954214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:26.984215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f688a2-c6e4-457d-ae97-48118b83f126 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.018292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.018868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.019157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.019437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.032691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:27.037701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:27.038605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:27.039570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 30/69] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:27.040206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.053511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e8f0eb-c72b-4afb-8be3-65f568baa7b7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.088294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.088910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.089197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.089517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.090389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:27.090497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.091858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.096284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.100148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 40/70] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.100977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.139391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c7a6e8-0c53-418b-afe9-564c660d8159 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.151573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.174512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.175137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.175421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.175695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.176811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.178306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.179902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 31/71] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.180665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.202397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbff7cf-9d2f-4a12-b0a1-5cf201256751 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.237251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.237943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.238228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.238504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.239203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:27.239512 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.240900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.248918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 41/72] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.254979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.284407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ceb6c-c902-45c5-994b-8fe5a4ab6009 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.308508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.319570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.320358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.320645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.320927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] PATH_INFO: `/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.334030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] RBAC: Authorizing `identity:get_user(user_id=2e21a1848e62493eb1c08ca9a1bdbcc6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.335509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.338931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 32/73] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.344268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.362057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e98eeda-c485-4109-8234-e410f674eb42 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.372923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.396179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.396805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.397096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.397428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.398099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:27.399035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.400466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.408937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 42/74] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.409819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.448713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d8046c-52d6-456e-a885-c29214de11b0 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.491842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.493048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.493351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.493642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] PATH_INFO: `/v3/users/8481404e88e44e68b2c70669768eebe2` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.494785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] RBAC: Authorizing `identity:get_user(user_id=8481404e88e44e68b2c70669768eebe2)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.496260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.498053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 33/75] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/users/8481404e88e44e68b2c70669768eebe2 => generated 247 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.498622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.514066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5683caf5-d724-4d75-8f75-e7d6cce23534 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.547696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.549686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.550306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.550604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.550891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.551544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.553091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.595636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce0b828-1a77-4943-ade9-494a2e7e562a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.629134 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.629795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.630084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.630362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] PATH_INFO: `/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.631493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] RBAC: Authorizing `identity:get_user(user_id=2e21a1848e62493eb1c08ca9a1bdbcc6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.633081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.634690 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 34/76] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6 => generated 275 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.635456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.650816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 43/77] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:46:27 2022] POST /identity/v3/users => generated 289 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:27.651737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.687839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.764569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d2d851-ee34-42b4-8883-007acd3282bb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.769898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2e4e24-d502-42d6-87c0-1abcc3e5770a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.786165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.806641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.807225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.807515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.807794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.808970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.810692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.810816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.811352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.811638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.811981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] PATH_INFO: `/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.812663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 35/78] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.813287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] RBAC: Authorizing `identity:get_user(user_id=2e21a1848e62493eb1c08ca9a1bdbcc6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.814873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:27.816508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 44/79] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6 => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:27.822564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.822737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:27.936051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:27.942301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54be2b0a-42c2-42ef-a726-76b3c39445fa None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.961836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289c8f58-3ac6-4821-a08b-542ac6eed9fa None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:27.991162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:27.992785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:27.993734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:27.994254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:27.996442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:27.998610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:28.000758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 45/80] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:28.001769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:28.003240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:28.003968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:28.004330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:28.004693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:28.006265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:28.009556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:28.011355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 36/81] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:27 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:28.012218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:28.092217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:28.127028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def5d9f-35c3-47f9-9fdc-8184b01b8637 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:28.153617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2f2683-67db-4a24-80cc-f8eb45e768cd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:28.173883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:28.174571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:28.174863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:28.175149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/2e21a1848e62493eb1c08ca9a1bdbcc6/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:28.177627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] RBAC: Authorizing `identity:create_grant(project_id=4d04c8e059934c64b4f015cf4a0e6641, user_id=2e21a1848e62493eb1c08ca9a1bdbcc6, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:28.179432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:28.193350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:28.193550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:28.193828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:28.194119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] PATH_INFO: `/v3/projects/326800bab4c044389b57e069661c052e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:28.195993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] RBAC: Authorizing `identity:get_project(project_id=326800bab4c044389b57e069661c052e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:28.197957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:28.199866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 37/82] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:28 2022] GET /identity/v3/projects/326800bab4c044389b57e069661c052e => generated 312 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:28.200361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:28.250937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 46/83] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:28 2022] PUT /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/2e21a1848e62493eb1c08ca9a1bdbcc6/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:28.259019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:28.312084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:28.352928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de21eed0-c00d-4237-80a5-c823c7d6a6cc None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:28.395184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:28.402405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:28.403127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:28.403415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:28.403699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/2e21a1848e62493eb1c08ca9a1bdbcc6/roles/335ad63ce3214c9aa7e4f4f058f97465` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:28.411705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=4d04c8e059934c64b4f015cf4a0e6641, user_id=2e21a1848e62493eb1c08ca9a1bdbcc6, role_id=335ad63ce3214c9aa7e4f4f058f97465)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:28.413295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:28.465550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dac314-95f8-44d7-96da-fa5592fddbc0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:28.502691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029eb2a5-8f7f-426c-8ff4-69adf952d08c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:28.503380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029eb2a5-8f7f-426c-8ff4-69adf952d08c None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:28.503668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029eb2a5-8f7f-426c-8ff4-69adf952d08c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:28.503949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029eb2a5-8f7f-426c-8ff4-69adf952d08c None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/8481404e88e44e68b2c70669768eebe2/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:28.505947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029eb2a5-8f7f-426c-8ff4-69adf952d08c None admin] RBAC: Authorizing `identity:create_grant(project_id=4d04c8e059934c64b4f015cf4a0e6641, user_id=8481404e88e44e68b2c70669768eebe2, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:28.507421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029eb2a5-8f7f-426c-8ff4-69adf952d08c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:28.519426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 38/84] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:28 2022] PUT /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/2e21a1848e62493eb1c08ca9a1bdbcc6/roles/335ad63ce3214c9aa7e4f4f058f97465 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:28.520254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:28.576893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 47/85] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:28 2022] PUT /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/8481404e88e44e68b2c70669768eebe2/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:28.642060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b97c773-31f7-4748-9478-3b11287372f0 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:28.680311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc29a8b-7c4c-4ef6-b449-22fa4697555a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:28.680975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc29a8b-7c4c-4ef6-b449-22fa4697555a None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:28.681262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc29a8b-7c4c-4ef6-b449-22fa4697555a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:28.681574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc29a8b-7c4c-4ef6-b449-22fa4697555a None admin] PATH_INFO: `/v3/projects/326800bab4c044389b57e069661c052e/users/2e21a1848e62493eb1c08ca9a1bdbcc6/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:28.683730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc29a8b-7c4c-4ef6-b449-22fa4697555a None admin] RBAC: Authorizing `identity:create_grant(project_id=326800bab4c044389b57e069661c052e, user_id=2e21a1848e62493eb1c08ca9a1bdbcc6, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:28.685517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc29a8b-7c4c-4ef6-b449-22fa4697555a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:28.784111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 39/86] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:28 2022] PUT /identity/v3/projects/326800bab4c044389b57e069661c052e/users/2e21a1848e62493eb1c08ca9a1bdbcc6/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:30.266123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045442e3-6c6f-4bcb-b433-1ac0cc47e4e3 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:30.266683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045442e3-6c6f-4bcb-b433-1ac0cc47e4e3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:30.266683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045442e3-6c6f-4bcb-b433-1ac0cc47e4e3 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:30.268041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 48/87] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:30.279031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0acd1ef-9282-48ae-aac1-56be546e76c0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:30.279475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0acd1ef-9282-48ae-aac1-56be546e76c0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:30.279726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0acd1ef-9282-48ae-aac1-56be546e76c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:30.306500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e0acd1ef-9282-48ae-aac1-56be546e76c0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:30.307753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e0acd1ef-9282-48ae-aac1-56be546e76c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:30.350035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:30.386703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 40/88] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:30.841140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac45a7ac-4f44-423f-b765-86e4c58915b2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:30.841530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac45a7ac-4f44-423f-b765-86e4c58915b2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:30.841741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac45a7ac-4f44-423f-b765-86e4c58915b2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:30.843101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 49/89] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:30.851855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7f74-ebda-4452-9ff0-497ce2d70fdd None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:30.852147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7f74-ebda-4452-9ff0-497ce2d70fdd None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:30.852426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7f74-ebda-4452-9ff0-497ce2d70fdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:30.875322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b9676c-3c90-4454-ba31-ed688065272c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:30.875631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b9676c-3c90-4454-ba31-ed688065272c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:30.875898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b9676c-3c90-4454-ba31-ed688065272c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:30.877552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 50/90] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:30.883207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bb6817-3abf-4d2d-986f-4f1703ff115d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:30.883458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bb6817-3abf-4d2d-986f-4f1703ff115d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:30.883829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bb6817-3abf-4d2d-986f-4f1703ff115d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:30.913879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-bd6b7f74-ebda-4452-9ff0-497ce2d70fdd None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:30.914810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a1bb6817-3abf-4d2d-986f-4f1703ff115d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:30.915094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bd6b7f74-ebda-4452-9ff0-497ce2d70fdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:30.916031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a1bb6817-3abf-4d2d-986f-4f1703ff115d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:31.028952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 41/91] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:31.030481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6b7f74-ebda-4452-9ff0-497ce2d70fdd None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:31.067230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 51/92] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:31.137171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6b7f74-ebda-4452-9ff0-497ce2d70fdd None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:31.138833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:31.174296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:31.174945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:31.175237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:31.175518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:31.186764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:31.191787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:31.192626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:31.193416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:31.197157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 42/93] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:31.200307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bb6817-3abf-4d2d-986f-4f1703ff115d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:31.217757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:31.321579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bb6817-3abf-4d2d-986f-4f1703ff115d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:31.337402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:31.405641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:31.406204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:31.406493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:31.406776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:31.407456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:31.407765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:31.409167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:31.428707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 52/94] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:31.450935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:31.485309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67074d4d-5bad-4823-8f83-e87069a184e3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:31.530301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:31.530956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:31.531244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:31.531528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:31.532664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:31.534434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:31.536255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 43/95] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:31.543569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:31.580179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:31.665767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:31.667784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812dc5ef-025d-4e0c-a7c5-3cddb11c300b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:31.710187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:31.712069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:31.712366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:31.712663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:31.713341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:31.714832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:31.739292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c0b8f-c56e-4593-b1cf-2716628b8718 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:31.776260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:31.776911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:31.777200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:31.777510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] PATH_INFO: `/v3/users/37af379683c849a6b3ce37af792cc986` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:31.794500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] RBAC: Authorizing `identity:get_user(user_id=37af379683c849a6b3ce37af792cc986)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:31.796129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:31.798427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 53/96] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:31 2022] GET /identity/v3/users/37af379683c849a6b3ce37af792cc986 => generated 289 bytes in 348 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:31.799507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:31.809812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 44/97] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:46:31 2022] POST /identity/v3/projects => generated 302 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:31.816239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:31.927548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9f4049-3e68-4d0c-af04-e4de2cb29159 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:31.969495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:31.970406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:31.970771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:31.971120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:31.978402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:31.984935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:31.990996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:31.996191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:31.996940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 54/98] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:31.999592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:32.110969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:32.123031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653d8d9-e8dc-467a-afd8-04edfd025d23 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:32.164565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:32.165349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.165818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.166165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.166843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:32.167148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:32.168539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:32.178943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 55/99] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:32.185277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:32.186138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df5bf17-c673-4493-9f14-8977bd6fecbf None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:32.256003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:32.270010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f93b22-eedd-4ea6-8bb1-269cf5e003e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:32.271894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f93b22-eedd-4ea6-8bb1-269cf5e003e5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.271894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f93b22-eedd-4ea6-8bb1-269cf5e003e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.271894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f93b22-eedd-4ea6-8bb1-269cf5e003e5 None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.272995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f93b22-eedd-4ea6-8bb1-269cf5e003e5 None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:32.276262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f93b22-eedd-4ea6-8bb1-269cf5e003e5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:32.279328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 45/100] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:31 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 464 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:32.280528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a136bb-22fc-4b98-8789-229f48536700 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.280817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a136bb-22fc-4b98-8789-229f48536700 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.281100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a136bb-22fc-4b98-8789-229f48536700 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.284178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 46/101] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:32.285020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a136bb-22fc-4b98-8789-229f48536700 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:32.328849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddf6b95-e6a4-4c95-8c51-7dcaedd21fa8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:32.370254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0247ad5-849e-4b91-8c30-911609b240da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:32.370898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0247ad5-849e-4b91-8c30-911609b240da None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.371189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0247ad5-849e-4b91-8c30-911609b240da None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.371471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0247ad5-849e-4b91-8c30-911609b240da None admin] PATH_INFO: `/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.372608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0247ad5-849e-4b91-8c30-911609b240da None admin] RBAC: Authorizing `identity:get_user(user_id=2e21a1848e62493eb1c08ca9a1bdbcc6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:32.374109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0247ad5-849e-4b91-8c30-911609b240da None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:32.375715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 56/102] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:32 2022] GET /identity/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6 => generated 275 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:32.376835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485803b-5ecb-41a1-af4c-ea559646b595 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.377122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485803b-5ecb-41a1-af4c-ea559646b595 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.381352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485803b-5ecb-41a1-af4c-ea559646b595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.398820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4485803b-5ecb-41a1-af4c-ea559646b595 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:32.400128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4485803b-5ecb-41a1-af4c-ea559646b595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:32.418269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a136bb-22fc-4b98-8789-229f48536700 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:32.454300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f5620-903d-4233-8031-4296645e4978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:32.454970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f5620-903d-4233-8031-4296645e4978 None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.455263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f5620-903d-4233-8031-4296645e4978 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.455539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f5620-903d-4233-8031-4296645e4978 None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/37af379683c849a6b3ce37af792cc986/roles/e7600b54d8dc4d2d9a7c5895b14e1f6d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.457501 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0f5620-903d-4233-8031-4296645e4978 None admin] RBAC: Authorizing `identity:create_grant(project_id=4d04c8e059934c64b4f015cf4a0e6641, user_id=37af379683c849a6b3ce37af792cc986, role_id=e7600b54d8dc4d2d9a7c5895b14e1f6d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:32.458964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0f5620-903d-4233-8031-4296645e4978 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:32.475299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 57/103] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:32.477032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8235b44-87bd-4864-ac84-4a620f8a809a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.477341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8235b44-87bd-4864-ac84-4a620f8a809a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.477652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8235b44-87bd-4864-ac84-4a620f8a809a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.481389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 58/104] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:32.481389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8235b44-87bd-4864-ac84-4a620f8a809a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:32.546640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:32.551481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 47/105] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:32 2022] PUT /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/users/37af379683c849a6b3ce37af792cc986/roles/e7600b54d8dc4d2d9a7c5895b14e1f6d => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:32.553709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f4fcfc-7d5f-4767-8f24-b849be5fb983 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.555135 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f4fcfc-7d5f-4767-8f24-b849be5fb983 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.555829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f4fcfc-7d5f-4767-8f24-b849be5fb983 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.557621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 48/106] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:32.563395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f86b22-a199-44f1-b2a4-183fd6640916 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.563765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f86b22-a199-44f1-b2a4-183fd6640916 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.564081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f86b22-a199-44f1-b2a4-183fd6640916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.585990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-44f86b22-a199-44f1-b2a4-183fd6640916 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:32.587818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-44f86b22-a199-44f1-b2a4-183fd6640916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:32.617521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8235b44-87bd-4864-ac84-4a620f8a809a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:32.625067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:32.667276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:32.668351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.668873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.669427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.671762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 49/107] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:32.673088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831845da-33b3-4c52-add1-fa6d49fe8982 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:32.673415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831845da-33b3-4c52-add1-fa6d49fe8982 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:32.673706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831845da-33b3-4c52-add1-fa6d49fe8982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:32.722961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-831845da-33b3-4c52-add1-fa6d49fe8982 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:32.723691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-831845da-33b3-4c52-add1-fa6d49fe8982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:32.809601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:32.811219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:32.865862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 50/108] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:32.866716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831845da-33b3-4c52-add1-fa6d49fe8982 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:32.889709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:32.946693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 59/109] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:32.952188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:32.957107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:32.991632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831845da-33b3-4c52-add1-fa6d49fe8982 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.029569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.030025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.030386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.030734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.042400 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:33.048252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:33.049102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:33.049896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 51/110] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:33.053356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.158688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c0a316-2874-4916-92b0-22259cb762bf None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.196169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.196869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.197157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.197505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.210017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:33.216065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:33.216903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:33.217714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:33.218284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 52/111] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:33.218830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.223575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:33.259235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff80c7-a095-45dc-871d-986ee62f909f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.304745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.305837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.306126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.306404 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.307561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:33.309752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:33.312152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 60/112] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:32 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:33.317577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.336392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07adcc8d-069e-41c0-8419-27aaa826c215 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.342909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:33.377052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.377856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.378202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.378486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.379155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:33.379524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:33.381068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:33.390680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 53/113] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:33.391510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.428929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354d550-127f-4caa-8069-26b9c127824e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.458079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:33.466143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.476296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.476296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.476296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.476296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:33.476296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:33.476296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:33.478671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 61/114] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:33.479498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.512778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a68bf08-4450-4312-9452-0739e5e4eb2c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.558685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.559597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.559885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.560161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.571416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:33.571898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:33.577109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:33.579145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 54/115] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:33.580559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.588899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179c38d7-28c8-4e16-8ba5-1ed1f6ae66a2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.626665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.627384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.627672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.627947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.628781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] RBAC: Adding query filter params (role.id=f1f68385aeb8421b80010aaf19d20d6e, scope.project.id=4d04c8e059934c64b4f015cf4a0e6641, user.id=2e21a1848e62493eb1c08ca9a1bdbcc6) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:33.629095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:33.630578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:33.637624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 62/116] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/role_assignments?user.id=2e21a1848e62493eb1c08ca9a1bdbcc6&scope.project.id=4d04c8e059934c64b4f015cf4a0e6641&role.id=f1f68385aeb8421b80010aaf19d20d6e => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:33.638434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.691471 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46dabdc-1e3f-468b-a0fd-1d5abad02b96 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.707331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:33.731780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.734581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.734581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.734581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] PATH_INFO: `/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.734581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] RBAC: Authorizing `identity:get_user(user_id=2e21a1848e62493eb1c08ca9a1bdbcc6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:33.736009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:33.738036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 55/117] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:33.738933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.860912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4cd0e0-51cb-4cab-b85f-025b46f7dcf8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.865668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e3d04-9bb0-454a-8642-18be0c7952b5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:33.936790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.937479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.937907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.938164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] PATH_INFO: `/v3/users/8481404e88e44e68b2c70669768eebe2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.938972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:33.939416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] RBAC: Authorizing `identity:get_user(user_id=8481404e88e44e68b2c70669768eebe2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:33.939587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:33.939867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:33.940142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:33.940816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:33.941009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:33.941161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:33.942519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:33.942973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 63/118] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/users/8481404e88e44e68b2c70669768eebe2 => generated 247 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:33.943469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.956053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 56/119] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:33.957519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:33.971301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.023348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.071738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6576004-6a1f-47d0-b4e3-32f013f09b0f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.085580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f43bf2-8184-48a2-a18c-30592c73b256 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.108597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:34.109224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.109538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.109818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.125996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:34.126547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.126835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.127119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.188198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:34.189915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:34.203989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:34.205592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:34.208986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.251148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.302196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 64/120] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:34.303381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:34.315030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 57/121] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:34.315978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:34.416178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58411c38-d9fc-44d6-86fd-db39e23dd222 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.427090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7a7dbb-0e09-4fab-8059-8c8e9be1b6fc None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.435789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.455729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:34.456638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.457085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.457530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] PATH_INFO: `/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.458953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] RBAC: Authorizing `identity:get_user(user_id=2e21a1848e62493eb1c08ca9a1bdbcc6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:34.460740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:34.462513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 65/122] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:34 2022] GET /identity/v3/users/2e21a1848e62493eb1c08ca9a1bdbcc6 => generated 275 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:34.463426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:34.463931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:34.464649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.464971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.465225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] PATH_INFO: `/v3/projects/326800bab4c044389b57e069661c052e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.466442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] RBAC: Authorizing `identity:get_project(project_id=326800bab4c044389b57e069661c052e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:34.468125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:34.469925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 58/123] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:34 2022] GET /identity/v3/projects/326800bab4c044389b57e069661c052e => generated 312 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:34.470625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:34.483824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.576094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6950a-19a9-4498-875e-7af25b4749c2 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.580276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059aae9b-761c-4a03-9cab-8a825a99b9bd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.611289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8712ab0a-455c-4f54-bf79-9b931dd1ced8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:34.611957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8712ab0a-455c-4f54-bf79-9b931dd1ced8 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.612221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8712ab0a-455c-4f54-bf79-9b931dd1ced8 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.612503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8712ab0a-455c-4f54-bf79-9b931dd1ced8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.625279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:34.626516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.626815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.627100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.628744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:34.630706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:34.632897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 66/124] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:34 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:34.634476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:34.705271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.707176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8712ab0a-455c-4f54-bf79-9b931dd1ced8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:34.709528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8712ab0a-455c-4f54-bf79-9b931dd1ced8 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:34.725729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.759043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ab631b-821f-4bd6-ba34-e43b7864887f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.799763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:34.800520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.800810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.801094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.801854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] RBAC: Adding query filter params (role.id=f1f68385aeb8421b80010aaf19d20d6e, scope.project.id=326800bab4c044389b57e069661c052e, user.id=2e21a1848e62493eb1c08ca9a1bdbcc6) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:34.802169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:34.803650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:34.810188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 67/125] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:34 2022] GET /identity/v3/role_assignments?user.id=2e21a1848e62493eb1c08ca9a1bdbcc6&scope.project.id=326800bab4c044389b57e069661c052e&role.id=f1f68385aeb8421b80010aaf19d20d6e => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:34.810188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:34.844865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 59/126] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:34.846378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45235e6-98e8-41a6-abfe-3e525bed609a None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:34.846677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45235e6-98e8-41a6-abfe-3e525bed609a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:34.846964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45235e6-98e8-41a6-abfe-3e525bed609a None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:34.848584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 60/127] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:34.850071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45235e6-98e8-41a6-abfe-3e525bed609a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:34.958004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341e3cb-40f1-4fa2-848d-a35a35733c54 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:34.984999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:34.994121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45235e6-98e8-41a6-abfe-3e525bed609a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:35.012827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:35.028092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24df3ee-151a-4971-9842-6e067a4a8d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:35.029132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24df3ee-151a-4971-9842-6e067a4a8d5b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.029698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24df3ee-151a-4971-9842-6e067a4a8d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.030177 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24df3ee-151a-4971-9842-6e067a4a8d5b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.031179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24df3ee-151a-4971-9842-6e067a4a8d5b None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=4d04c8e059934c64b4f015cf4a0e6641, user.id=8481404e88e44e68b2c70669768eebe2) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:35.032224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24df3ee-151a-4971-9842-6e067a4a8d5b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:35.035853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24df3ee-151a-4971-9842-6e067a4a8d5b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:35.040952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 68/128] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:34 2022] GET /identity/v3/role_assignments?user.id=8481404e88e44e68b2c70669768eebe2&scope.project.id=4d04c8e059934c64b4f015cf4a0e6641&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:35.042502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea95264-ee95-4506-8bbc-c00d4bb5b8db None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.042985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea95264-ee95-4506-8bbc-c00d4bb5b8db None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.043313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea95264-ee95-4506-8bbc-c00d4bb5b8db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.057174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:35.058359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.058359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.058544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.062309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:35.063750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:35.065506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 61/129] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:34 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:35.067315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-aea95264-ee95-4506-8bbc-c00d4bb5b8db None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:35.068962 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aea95264-ee95-4506-8bbc-c00d4bb5b8db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:35.075237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:35.159217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 69/130] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:35.160536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8097b4-9341-45a1-8dee-6c8ed9fe4975 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.160826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8097b4-9341-45a1-8dee-6c8ed9fe4975 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.161108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8097b4-9341-45a1-8dee-6c8ed9fe4975 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.162605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 70/131] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:35.179053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60633a12-92c5-4939-9fda-3cc30c0230b9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.180046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60633a12-92c5-4939-9fda-3cc30c0230b9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.180212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60633a12-92c5-4939-9fda-3cc30c0230b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.199850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf7d5d4-9cde-4df5-b606-a6226d94f090 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:35.200075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-60633a12-92c5-4939-9fda-3cc30c0230b9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:35.201072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-60633a12-92c5-4939-9fda-3cc30c0230b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:35.239848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:35.240815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.241336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.241665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.242993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] RBAC: Adding query filter params (role.id=335ad63ce3214c9aa7e4f4f058f97465, scope.project.id=4d04c8e059934c64b4f015cf4a0e6641, user.id=2e21a1848e62493eb1c08ca9a1bdbcc6) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:35.243314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:35.244915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:35.253069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 62/132] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:35 2022] GET /identity/v3/role_assignments?user.id=2e21a1848e62493eb1c08ca9a1bdbcc6&scope.project.id=4d04c8e059934c64b4f015cf4a0e6641&role.id=335ad63ce3214c9aa7e4f4f058f97465 => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:35.279930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 71/133] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:35.461887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:35.506238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:35.561112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60633a12-92c5-4939-9fda-3cc30c0230b9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:35.565412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ccf09e-5645-404e-abbc-31c5b850760c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:35.598403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:35.599038 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.599331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.599602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.600729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:35.601545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:35.602270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:35.603893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 63/134] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:35.609579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:35.658947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60633a12-92c5-4939-9fda-3cc30c0230b9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:35.692819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:35.693496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.693782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.694059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.704873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:35.709479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f889bc97-02da-4c58-82bf-a2d4bb9ff692 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:35.709831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:35.711333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:35.714001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 72/135] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:35 2022] GET /identity/v3/roles/reader => generated 84 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:35.717403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:35.720478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:35.743730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:35.744540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.744888 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.745178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.745863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:35.747360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:35.814792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:35.817537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98435907-418b-4123-819e-5a3129f51fe1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:35.827679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 64/136] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:46:35 2022] POST /identity/v3/users => generated 283 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:46:35.854414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:35.855055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:35.855341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:35.855618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:35.856413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:35.856722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:35.858189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:35.872656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 73/137] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:35 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:35.873915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:35.981406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0fd8c6-08a9-428c-8fb4-9242e38e3373 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:36.017043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:36.017716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:36.018006 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:36.018285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] PATH_INFO: `/v3/users/37af379683c849a6b3ce37af792cc986` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:36.019409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] RBAC: Authorizing `identity:get_user(user_id=37af379683c849a6b3ce37af792cc986)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:36.020881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:36.022487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 65/138] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:35 2022] GET /identity/v3/users/37af379683c849a6b3ce37af792cc986 => generated 289 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:36.028630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:36.135511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac329f2c-9c72-460f-9753-dfd38c35f427 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:36.170430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:36.171157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:36.171615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:36.171965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:36.205772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:36.247410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:36.248875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:36.355707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 74/139] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:36.361127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:36.397405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:36.465241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f78a65d-0545-4700-a6f3-fd769ec5c30f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:36.500566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6cca74-813d-44b3-9b50-bd5746bd86db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:36.501207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6cca74-813d-44b3-9b50-bd5746bd86db None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:36.501545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6cca74-813d-44b3-9b50-bd5746bd86db None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:36.501823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6cca74-813d-44b3-9b50-bd5746bd86db None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:36.502963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6cca74-813d-44b3-9b50-bd5746bd86db None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:36.504413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6cca74-813d-44b3-9b50-bd5746bd86db None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:36.506066 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 66/140] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:36 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:36.512717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:36.576893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:36.621234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a27e89-e57e-4814-8bfe-3de806eddea8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:36.662501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ed8a6-7a77-4e08-95af-d05d97e87ae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:36.663039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ed8a6-7a77-4e08-95af-d05d97e87ae8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:36.663318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ed8a6-7a77-4e08-95af-d05d97e87ae8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:36.663622 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ed8a6-7a77-4e08-95af-d05d97e87ae8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:36.664360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ed8a6-7a77-4e08-95af-d05d97e87ae8 None admin] RBAC: Adding query filter params (role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d, scope.project.id=4d04c8e059934c64b4f015cf4a0e6641, user.id=37af379683c849a6b3ce37af792cc986) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:36.664663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ed8a6-7a77-4e08-95af-d05d97e87ae8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:36.666138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ed8a6-7a77-4e08-95af-d05d97e87ae8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:36.672759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 75/141] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:36 2022] GET /identity/v3/role_assignments?user.id=37af379683c849a6b3ce37af792cc986&scope.project.id=4d04c8e059934c64b4f015cf4a0e6641&role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:38.091923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9486ed2-3beb-4dae-a2f3-f6d457250bdf None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.092445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9486ed2-3beb-4dae-a2f3-f6d457250bdf None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.092519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9486ed2-3beb-4dae-a2f3-f6d457250bdf None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.094040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 67/142] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:38.099063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a7883-10ba-449d-9d39-8aee0eccbce2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.099446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a7883-10ba-449d-9d39-8aee0eccbce2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.099634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a7883-10ba-449d-9d39-8aee0eccbce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.119222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4f0a7883-10ba-449d-9d39-8aee0eccbce2 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:38.120401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f0a7883-10ba-449d-9d39-8aee0eccbce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:38.192824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 76/143] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:38.288050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a0facd-1017-4941-b0f6-4e157043a7a8 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.288344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a0facd-1017-4941-b0f6-4e157043a7a8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.289108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a0facd-1017-4941-b0f6-4e157043a7a8 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.290692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 68/144] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:38.295784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3102c84-ae78-4055-8325-e69044de9477 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.296086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3102c84-ae78-4055-8325-e69044de9477 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.296372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3102c84-ae78-4055-8325-e69044de9477 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.315516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b3102c84-ae78-4055-8325-e69044de9477 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:38.316741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b3102c84-ae78-4055-8325-e69044de9477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:38.372819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:38.389731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 77/145] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:38.443600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f4e9b9-dc93-45bb-8686-ee8409ce5c05 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.443948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f4e9b9-dc93-45bb-8686-ee8409ce5c05 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.444133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f4e9b9-dc93-45bb-8686-ee8409ce5c05 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.446220 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 69/146] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:38.451314 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49cf8a-ec2f-4964-82d5-cdd4dc287551 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.451617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49cf8a-ec2f-4964-82d5-cdd4dc287551 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.451894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49cf8a-ec2f-4964-82d5-cdd4dc287551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.465412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f4e9b9-dc93-45bb-8686-ee8409ce5c05 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:38.471403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4f49cf8a-ec2f-4964-82d5-cdd4dc287551 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:38.472740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f49cf8a-ec2f-4964-82d5-cdd4dc287551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:38.550531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 78/147] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:38.571855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f4e9b9-dc93-45bb-8686-ee8409ce5c05 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:38.573598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:38.607439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:38.608079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.608369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.608653 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.619618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:38.624465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:38.626085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:38.626940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 70/148] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:46:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:38.631366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49cf8a-ec2f-4964-82d5-cdd4dc287551 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:38.654417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:38.729910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49cf8a-ec2f-4964-82d5-cdd4dc287551 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:38.745813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:38.759606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f4667-60d3-42b7-8489-8f0888f61fd8 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:38.763643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:38.764292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.764585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.764843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.765521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:38.765821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:38.767188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:38.780632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 79/149] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:38.781351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:38.797591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:38.798552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.798834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.799313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.814043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:38.815424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:38.816337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:38.817104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 71/150] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:46:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:38.823141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:38.847521 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:38.888120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1220794f-5512-4a54-b7c6-4b89185e977a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:38.921970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:38.922605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.922731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b7dd5f-0fbe-443d-96c4-1b5000b5d3a1 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:38.922891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.923168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] PATH_INFO: `/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.937858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] RBAC: Authorizing `identity:get_user(user_id=b82e6d6e2e2f4b31b019a4842f4aa1c3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:38.939483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:38.941127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 80/151] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:38 2022] GET /identity/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3 => generated 283 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:38.942019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:38.964092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:38.964730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:38.965015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:38.965289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:38.966497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:38.967966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:38.969555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 72/152] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:38.970312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:38.979120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:39.047872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04f3c35-c11d-4415-9f01-8f028b4dfd0f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:39.065750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:39.075838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41f66e-30ca-4b23-bc10-7ac0eef16e0e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:39.082250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:39.082865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:39.083149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:39.083424 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:39.084080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:39.084383 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:39.085925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:39.093822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 81/153] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:46:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:39.094596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:39.113300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:39.113968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:39.114261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:39.114583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:39.115713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:39.117857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:39.119628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 73/154] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:38 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:39.120322 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:39.190421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:39.194234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c756feb9-d1d9-435f-93f3-5fee6ea0181d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:39.221796 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b300d0-43b7-42aa-bfbb-56d56c07ca5e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:39.227812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:39.228430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:39.228713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:39.228989 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:39.229677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:39.231144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:39.256736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:39.257928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:39.258237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:39.258569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] PATH_INFO: `/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:39.259692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] RBAC: Authorizing `identity:get_user(user_id=b82e6d6e2e2f4b31b019a4842f4aa1c3)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:39.261134 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:39.262734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 74/155] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:39 2022] GET /identity/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3 => generated 283 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:39.263465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:39.289217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:39.308562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 82/156] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:46:39 2022] POST /identity/v3/users => generated 297 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:39.309950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:39.377626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225a78c7-648d-4044-a150-0bf28e78ca9a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:39.424538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f9ee2-dccf-45ff-a65f-d6d1486fc74e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:39.425369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f9ee2-dccf-45ff-a65f-d6d1486fc74e None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:39.425651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f9ee2-dccf-45ff-a65f-d6d1486fc74e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:39.425939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f9ee2-dccf-45ff-a65f-d6d1486fc74e None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/b82e6d6e2e2f4b31b019a4842f4aa1c3/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:39.427928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f9ee2-dccf-45ff-a65f-d6d1486fc74e None admin] RBAC: Authorizing `identity:create_grant(project_id=c9cd84395d7f49969b6df473fd521e6d, user_id=b82e6d6e2e2f4b31b019a4842f4aa1c3, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:39.428281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21aabbc-aeea-474d-9b97-1b136aed1886 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:39.430809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f9ee2-dccf-45ff-a65f-d6d1486fc74e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:39.466707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:39.467414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:39.467698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:39.467971 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:39.469127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:39.470654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:39.472332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 83/157] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:39 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:39.479725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:39.503880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 75/158] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:39 2022] PUT /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/b82e6d6e2e2f4b31b019a4842f4aa1c3/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:39.507559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:39.603815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64b38be-bbdb-4298-9a1f-f1d2129bd382 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:39.641543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a15c8d3-85f9-4afd-a9f3-f8c7a16bd9a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:39.642149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a15c8d3-85f9-4afd-a9f3-f8c7a16bd9a7 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:39.642430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a15c8d3-85f9-4afd-a9f3-f8c7a16bd9a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:39.642703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a15c8d3-85f9-4afd-a9f3-f8c7a16bd9a7 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/b82e6d6e2e2f4b31b019a4842f4aa1c3/roles/335ad63ce3214c9aa7e4f4f058f97465` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:39.644959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a15c8d3-85f9-4afd-a9f3-f8c7a16bd9a7 None admin] RBAC: Authorizing `identity:create_grant(project_id=c9cd84395d7f49969b6df473fd521e6d, user_id=b82e6d6e2e2f4b31b019a4842f4aa1c3, role_id=335ad63ce3214c9aa7e4f4f058f97465)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:39.646553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a15c8d3-85f9-4afd-a9f3-f8c7a16bd9a7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:39.708772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 84/159] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:39 2022] PUT /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/b82e6d6e2e2f4b31b019a4842f4aa1c3/roles/335ad63ce3214c9aa7e4f4f058f97465 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:46:41.700317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bfc767-c647-4c6a-9cd1-a8ef3212cf8e None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.700891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bfc767-c647-4c6a-9cd1-a8ef3212cf8e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.700891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bfc767-c647-4c6a-9cd1-a8ef3212cf8e None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.702255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 76/160] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:41.707596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9771c1f4-a037-4e6a-bc9e-ee7fcc86bb37 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.708010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9771c1f4-a037-4e6a-bc9e-ee7fcc86bb37 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.708178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9771c1f4-a037-4e6a-bc9e-ee7fcc86bb37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.722367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49741e8a-691b-4947-901e-b30102a03b7d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.722929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49741e8a-691b-4947-901e-b30102a03b7d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.723437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49741e8a-691b-4947-901e-b30102a03b7d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.725230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 77/161] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:41.728281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9771c1f4-a037-4e6a-bc9e-ee7fcc86bb37 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:41.729562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9771c1f4-a037-4e6a-bc9e-ee7fcc86bb37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:41.731312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7489543-6d1e-4673-a622-5e38f203396a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.731816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7489543-6d1e-4673-a622-5e38f203396a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.732302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7489543-6d1e-4673-a622-5e38f203396a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.751346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a7489543-6d1e-4673-a622-5e38f203396a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:41.752848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a7489543-6d1e-4673-a622-5e38f203396a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:41.803573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 85/162] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:41.805541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0c571f-cb39-41fe-a2f0-89dcc62d6275 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.806063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0c571f-cb39-41fe-a2f0-89dcc62d6275 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.806543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0c571f-cb39-41fe-a2f0-89dcc62d6275 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.808160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 86/163] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:41.814779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef05d57-cba4-4dd9-9051-f382c9823d5a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.815402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef05d57-cba4-4dd9-9051-f382c9823d5a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.815892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef05d57-cba4-4dd9-9051-f382c9823d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.837078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 78/164] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:41.841625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8ef05d57-cba4-4dd9-9051-f382c9823d5a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:41.841625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8ef05d57-cba4-4dd9-9051-f382c9823d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:41.848616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:41.911249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 87/165] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:41.934807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45714ec-46dd-460d-a3ea-cf3edd5765bd None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.935400 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45714ec-46dd-460d-a3ea-cf3edd5765bd None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.935868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45714ec-46dd-460d-a3ea-cf3edd5765bd None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.937448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 79/166] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:41.945754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d373826-c33d-4610-a3fe-42aee1f15d48 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:41.954315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d373826-c33d-4610-a3fe-42aee1f15d48 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:41.954315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d373826-c33d-4610-a3fe-42aee1f15d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:41.964906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6d373826-c33d-4610-a3fe-42aee1f15d48 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:41.966135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6d373826-c33d-4610-a3fe-42aee1f15d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:42.050042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 88/167] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:42.080094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45714ec-46dd-460d-a3ea-cf3edd5765bd None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.099851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.120465 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d373826-c33d-4610-a3fe-42aee1f15d48 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.205201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45714ec-46dd-460d-a3ea-cf3edd5765bd None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.212583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.242709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.243344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.243627 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.243897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] PATH_INFO: `/v3/roles/member` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.247294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d373826-c33d-4610-a3fe-42aee1f15d48 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.254128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:42.260301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:42.261231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:42.262111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 80/168] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:42.262806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.288339 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.289639 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.289919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.290295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.300897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:42.306613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:42.307475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:42.308266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 89/169] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:42.308927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.350345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.372757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b038799-3db4-43c1-a450-7a3553e29dea None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.408116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.408748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.409030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.409302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.409700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596da3cb-b287-4a5f-9f72-91a111a07450 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.410452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:42.411913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:42.413510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 81/170] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:42.416634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.441113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.446458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.447066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.447350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.447621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.448280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:42.448582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:42.449980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:42.457953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 90/171] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:42.458682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.515049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d7115e-7c5c-49ab-aad6-d634f0d411e5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.548940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.549577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.549863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.550143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.550813 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:42.551118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:42.552526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:42.557440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081d6b82-b6d7-4295-815a-4bffb60bdc74 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.561473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 82/172] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:42.564874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.573465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.591412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.602217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.602217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.602217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.602564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:42.607749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:42.608734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 91/173] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:42.610057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.665445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba6bca9-1913-4e63-83fd-d8d6d4c09268 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.667353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.699316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.699926 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.700209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.700482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.701132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:42.702619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:42.713211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ffda6d-3c7d-4ee6-a2ac-a305e33cb66d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.746645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.747276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.747560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.747837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] PATH_INFO: `/v3/users/1f1ed5408ece454e8587b6996231aaac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.761436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] RBAC: Authorizing `identity:get_user(user_id=1f1ed5408ece454e8587b6996231aaac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:42.762887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:42.764468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 92/174] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/users/1f1ed5408ece454e8587b6996231aaac => generated 297 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:42.765225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.816700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 83/175] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:46:42 2022] POST /identity/v3/users => generated 297 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:42.817773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.876420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372b3ee-fe7f-4584-a44d-fd77216df76b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.889840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.891491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:42.913848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.914513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.914800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.915074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] PATH_INFO: `/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.916237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] RBAC: Authorizing `identity:get_user(user_id=b82e6d6e2e2f4b31b019a4842f4aa1c3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:42.917745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:42.919416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 93/176] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3 => generated 283 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:42.920366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:42.940627 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7cb340-0aca-4ba9-bea0-5a3606bf984e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:42.981629 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:42.982275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:42.982634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:42.982980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:42.983857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:42.984119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:42.985635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:42.994498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 84/177] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:43.000628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:43.035389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a529f0-009a-4c54-b615-f6979ce32b41 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:43.076604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:43.077300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:43.077927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:43.077927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:43.079025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:43.080481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:43.082171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 94/178] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:43.082938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:43.129656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f2937b-e678-4895-804c-fe5acadea402 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:43.145664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:43.155650 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:43.171047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:43.171682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:43.171968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:43.172242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:43.218483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bec866-1b74-47c4-83b2-78152a45fb73 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:43.261207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:43.262635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:43.266330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:43.266985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:43.267266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:43.267538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] PATH_INFO: `/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:43.269606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] RBAC: Authorizing `identity:get_user(user_id=b82e6d6e2e2f4b31b019a4842f4aa1c3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:43.270845 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:43.272601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 95/179] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:43 2022] GET /identity/v3/users/b82e6d6e2e2f4b31b019a4842f4aa1c3 => generated 283 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:43.277488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:43.394621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee1bd1b-8513-4129-8760-652e20d9b6b8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:43.404272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 85/180] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:43.404872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:43.416273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:43.432299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:43.441657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:43.443327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:43.443811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:43.444278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/1f1ed5408ece454e8587b6996231aaac/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:43.447048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] RBAC: Authorizing `identity:create_grant(project_id=c9cd84395d7f49969b6df473fd521e6d, user_id=1f1ed5408ece454e8587b6996231aaac, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:43.448723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:43.508586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3fecd-05e2-4551-a165-f2243c46c253 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:43.529514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 96/181] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:43 2022] PUT /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/1f1ed5408ece454e8587b6996231aaac/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:43.529514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:43.563003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:43.563741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:43.564017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:43.564294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:43.660016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ef8a48-d696-4fb8-868e-42c9da49b01d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:43.703742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:43.704890 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:43.705403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:43.705875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:43.708197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:43.710023 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:43.712120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 97/182] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:43 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:43.721867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:43.729069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:43.730644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:43.764754 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:43.792859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:43.853518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 86/183] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:43.859491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:43.867248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d28588-11ee-462e-8c41-522aa212b450 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:43.911151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:43.912311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:43.912798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:43.913271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:43.915030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=c9cd84395d7f49969b6df473fd521e6d, user.id=b82e6d6e2e2f4b31b019a4842f4aa1c3) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:43.915592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:43.917547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:43.924530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 98/184] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:43 2022] GET /identity/v3/role_assignments?user.id=b82e6d6e2e2f4b31b019a4842f4aa1c3&scope.project.id=c9cd84395d7f49969b6df473fd521e6d&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:44.051330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307e8558-93b0-4270-a1d1-f01413a9e9de None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:44.084410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:44.108292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ee1900-c3dd-495d-885c-760adcf08a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:44.109407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee1900-c3dd-495d-885c-760adcf08a28 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:44.109939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee1900-c3dd-495d-885c-760adcf08a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:44.110407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee1900-c3dd-495d-885c-760adcf08a28 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:44.111831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ee1900-c3dd-495d-885c-760adcf08a28 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:44.113632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ee1900-c3dd-495d-885c-760adcf08a28 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:44.115875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 87/185] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:43 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:44.122182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:44.237098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2848d834-bb01-475b-ab18-b04629f04766 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:44.247009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:44.276831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6e7a14-59fd-4e4c-9dde-3721790809ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:44.277880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6e7a14-59fd-4e4c-9dde-3721790809ef None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:44.278417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6e7a14-59fd-4e4c-9dde-3721790809ef None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:44.278891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6e7a14-59fd-4e4c-9dde-3721790809ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:44.279832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6e7a14-59fd-4e4c-9dde-3721790809ef None admin] RBAC: Adding query filter params (role.id=335ad63ce3214c9aa7e4f4f058f97465, scope.project.id=c9cd84395d7f49969b6df473fd521e6d, user.id=b82e6d6e2e2f4b31b019a4842f4aa1c3) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:44.280368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6e7a14-59fd-4e4c-9dde-3721790809ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:44.282050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6e7a14-59fd-4e4c-9dde-3721790809ef None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:44.292778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 99/186] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:44 2022] GET /identity/v3/role_assignments?user.id=b82e6d6e2e2f4b31b019a4842f4aa1c3&scope.project.id=c9cd84395d7f49969b6df473fd521e6d&role.id=335ad63ce3214c9aa7e4f4f058f97465 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:45.300030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838e0b2f-9620-41ab-8732-5cdbe721afee None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:45.300516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838e0b2f-9620-41ab-8732-5cdbe721afee None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:45.300516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838e0b2f-9620-41ab-8732-5cdbe721afee None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:45.301899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 88/187] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:45.307507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9932f77e-c62d-46cc-9bfd-0a11f2064349 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:45.307889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9932f77e-c62d-46cc-9bfd-0a11f2064349 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:45.308066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9932f77e-c62d-46cc-9bfd-0a11f2064349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:45.326269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9932f77e-c62d-46cc-9bfd-0a11f2064349 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:45.327443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9932f77e-c62d-46cc-9bfd-0a11f2064349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:45.376114 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b7412-3735-4e2d-8b91-c10a2f50c0cf None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:45.376403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b7412-3735-4e2d-8b91-c10a2f50c0cf None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:45.376679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b7412-3735-4e2d-8b91-c10a2f50c0cf None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:45.378094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 89/188] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:45.383752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31686d9-2fb3-408e-b552-1b718bf10998 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:45.384115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31686d9-2fb3-408e-b552-1b718bf10998 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:45.384473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31686d9-2fb3-408e-b552-1b718bf10998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:45.399289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 100/189] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:45.405504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f31686d9-2fb3-408e-b552-1b718bf10998 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:45.405792 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f31686d9-2fb3-408e-b552-1b718bf10998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:45.474473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 90/190] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:45.668632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9932f77e-c62d-46cc-9bfd-0a11f2064349 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:45.750828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31686d9-2fb3-408e-b552-1b718bf10998 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:45.771425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9932f77e-c62d-46cc-9bfd-0a11f2064349 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:45.773141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:45.819393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:45.820091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:45.820369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:45.820641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:45.832032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:45.837050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:45.838016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:45.838802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:45.843059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 101/191] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:45.843864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:45.866866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31686d9-2fb3-408e-b552-1b718bf10998 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:45.891747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:45.904487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:45.905108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:45.905413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:45.905691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:45.907000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:45.908233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:45.910031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 91/192] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:45.915290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:45.955549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378500e7-93de-40bd-b971-eef73cddc0fb None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:45.988878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8582d2c3-6092-4ab2-b94f-2cbe19e09adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:45.989519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582d2c3-6092-4ab2-b94f-2cbe19e09adb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:45.989806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582d2c3-6092-4ab2-b94f-2cbe19e09adb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:45.990088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582d2c3-6092-4ab2-b94f-2cbe19e09adb None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:45.990748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582d2c3-6092-4ab2-b94f-2cbe19e09adb None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:45.991053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582d2c3-6092-4ab2-b94f-2cbe19e09adb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:45.992654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582d2c3-6092-4ab2-b94f-2cbe19e09adb None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:46.000968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 102/193] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:46.000968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f322b08e-9261-4606-9752-8ee4091a7673 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.000968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f322b08e-9261-4606-9752-8ee4091a7673 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.000968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f322b08e-9261-4606-9752-8ee4091a7673 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.000968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 103/194] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:46 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:46.006538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f322b08e-9261-4606-9752-8ee4091a7673 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:46.014793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a906d1-35f7-465f-97f4-d0c3e17423a3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:46.035291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:46.049833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb3243f-b5e5-4c8e-a5fd-e3831b94e80a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:46.050455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3243f-b5e5-4c8e-a5fd-e3831b94e80a None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.050735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3243f-b5e5-4c8e-a5fd-e3831b94e80a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.051008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3243f-b5e5-4c8e-a5fd-e3831b94e80a None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.051656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb3243f-b5e5-4c8e-a5fd-e3831b94e80a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:46.053098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb3243f-b5e5-4c8e-a5fd-e3831b94e80a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:46.107400 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 92/195] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:46:45 2022] POST /identity/v3/users => generated 293 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:46:46.108673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8604f75-47e8-416d-a1d9-d62db8457036 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.109001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8604f75-47e8-416d-a1d9-d62db8457036 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.109241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8604f75-47e8-416d-a1d9-d62db8457036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.111899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f322b08e-9261-4606-9752-8ee4091a7673 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:46.129337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c8604f75-47e8-416d-a1d9-d62db8457036 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:46.130644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c8604f75-47e8-416d-a1d9-d62db8457036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:46.153926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:46.154582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.154864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.155145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] PATH_INFO: `/v3/users/6f018fc8d39842848a4271aaed35ae83` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.172050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] RBAC: Authorizing `identity:get_user(user_id=6f018fc8d39842848a4271aaed35ae83)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:46.173762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:46.175828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 104/196] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:46 2022] GET /identity/v3/users/6f018fc8d39842848a4271aaed35ae83 => generated 297 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:46.181582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:46.182057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:46.214020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 93/197] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:46.284355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:46.304891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeac757e-505d-4859-b221-edbceaedb573 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:46.340443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:46.341190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.341435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.341714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.342841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:46.344277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:46.346142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 105/198] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:46 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:46.352540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8604f75-47e8-416d-a1d9-d62db8457036 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:46.459866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8604f75-47e8-416d-a1d9-d62db8457036 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:46.489742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:46.497444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:46.498126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.498406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.498679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/6f018fc8d39842848a4271aaed35ae83/roles/e7600b54d8dc4d2d9a7c5895b14e1f6d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.500755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] RBAC: Authorizing `identity:create_grant(project_id=c9cd84395d7f49969b6df473fd521e6d, user_id=6f018fc8d39842848a4271aaed35ae83, role_id=e7600b54d8dc4d2d9a7c5895b14e1f6d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:46.502243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:46.577885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 94/199] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:46:46 2022] PUT /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d/users/6f018fc8d39842848a4271aaed35ae83/roles/e7600b54d8dc4d2d9a7c5895b14e1f6d => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:46.600760 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e9e5b9-45ec-418b-90eb-6bc7877902b2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:46.642704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:46.643792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.644292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.644778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.658549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:46.664228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:46.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:46.665991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:46.668414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 106/200] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:46.674326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:46.715741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:46.794599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebaefe8-95be-4459-8893-3d6cc8ec6712 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:46.832564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:46.833181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.833498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.833773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.834442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:46.834752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:46.836356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:46.845538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 95/201] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:46.851829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:46.865296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:46.959993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5d3546-babb-49a2-897d-5b32b4cc7ed6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:46.997169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:46.997848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:46.998134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:46.998414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] PATH_INFO: `/v3/users/1f1ed5408ece454e8587b6996231aaac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:46.999574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] RBAC: Authorizing `identity:get_user(user_id=1f1ed5408ece454e8587b6996231aaac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:47.001039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:47.003028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 107/202] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:46 2022] GET /identity/v3/users/1f1ed5408ece454e8587b6996231aaac => generated 297 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:47.008528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:47.125777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:47.130587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c8780-b5f2-404a-83e9-f6aa4a79a3e8 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:47.168203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:47.169227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:47.169737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:47.170207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:47.246825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:47.248695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:47.352539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:47.358654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 96/203] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:47.364511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:47.446761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:47.469908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038e29a-164c-4295-a3c7-307ffd70c6d7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:47.511420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24e6e31-20fb-41d1-914c-9d70b53525d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:47.512064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e6e31-20fb-41d1-914c-9d70b53525d8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:47.512343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e6e31-20fb-41d1-914c-9d70b53525d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:47.512618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e6e31-20fb-41d1-914c-9d70b53525d8 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:47.513761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24e6e31-20fb-41d1-914c-9d70b53525d8 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:47.515191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24e6e31-20fb-41d1-914c-9d70b53525d8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:47.516805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 108/204] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:47 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:47.523508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:47.633608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4482444-d7a1-491d-9760-be5ae13e6b90 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:47.668783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6c4ce5-6648-41dc-97f1-0bc79c63cf08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:47.669425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6c4ce5-6648-41dc-97f1-0bc79c63cf08 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:47.669708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6c4ce5-6648-41dc-97f1-0bc79c63cf08 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:47.669979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6c4ce5-6648-41dc-97f1-0bc79c63cf08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:47.670707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6c4ce5-6648-41dc-97f1-0bc79c63cf08 None admin] RBAC: Adding query filter params (role.id=f1f68385aeb8421b80010aaf19d20d6e, scope.project.id=c9cd84395d7f49969b6df473fd521e6d, user.id=1f1ed5408ece454e8587b6996231aaac) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:47.671012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6c4ce5-6648-41dc-97f1-0bc79c63cf08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:47.672504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6c4ce5-6648-41dc-97f1-0bc79c63cf08 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:47.678794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 97/205] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:47 2022] GET /identity/v3/role_assignments?user.id=1f1ed5408ece454e8587b6996231aaac&scope.project.id=c9cd84395d7f49969b6df473fd521e6d&role.id=f1f68385aeb8421b80010aaf19d20d6e => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:48.483540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23027905-dc6a-476d-9bd4-7ed9b9ff1ca6 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:48.484088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23027905-dc6a-476d-9bd4-7ed9b9ff1ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:48.484088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23027905-dc6a-476d-9bd4-7ed9b9ff1ca6 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:48.485603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 109/206] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:48.490586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395b8f9b-8fee-466f-bcb3-c0bcaaaec7be None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:48.490986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395b8f9b-8fee-466f-bcb3-c0bcaaaec7be None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:48.491159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395b8f9b-8fee-466f-bcb3-c0bcaaaec7be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:48.509979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-395b8f9b-8fee-466f-bcb3-c0bcaaaec7be None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:48.511161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-395b8f9b-8fee-466f-bcb3-c0bcaaaec7be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:48.560669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd13858-ede6-459b-8ffe-cada8e61b61b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:48.561008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd13858-ede6-459b-8ffe-cada8e61b61b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:48.561217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd13858-ede6-459b-8ffe-cada8e61b61b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:48.562722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 110/207] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:48.567312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51819eb9-a461-4b85-b69e-0830c356bbcb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:48.567597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51819eb9-a461-4b85-b69e-0830c356bbcb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:48.567876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51819eb9-a461-4b85-b69e-0830c356bbcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:48.572295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:48.582303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 98/208] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:48.587243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-51819eb9-a461-4b85-b69e-0830c356bbcb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:48.588409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-51819eb9-a461-4b85-b69e-0830c356bbcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:48.655520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 111/209] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:48.852242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395b8f9b-8fee-466f-bcb3-c0bcaaaec7be None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:48.870325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a642ed46-3adf-4612-a5d7-7509814276e9 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:48.870647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a642ed46-3adf-4612-a5d7-7509814276e9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:48.870903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a642ed46-3adf-4612-a5d7-7509814276e9 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:48.872396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 112/210] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:48.877192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1263210c-43f4-4458-b4b4-8e9db479083d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:48.877515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1263210c-43f4-4458-b4b4-8e9db479083d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:48.877797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1263210c-43f4-4458-b4b4-8e9db479083d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:48.896194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1263210c-43f4-4458-b4b4-8e9db479083d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:48.897418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1263210c-43f4-4458-b4b4-8e9db479083d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:48.954433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395b8f9b-8fee-466f-bcb3-c0bcaaaec7be None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:48.967990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 113/211] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:48.968832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1263210c-43f4-4458-b4b4-8e9db479083d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:48.977007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:48.991925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:48.992576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:48.992959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:48.993123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.004155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:49.012313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:49.013448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:49.014367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:49.016271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 99/212] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:49.017176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:49.072709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:49.088779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1263210c-43f4-4458-b4b4-8e9db479083d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.121452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1955d2f-a5ca-433e-86ce-79c2bc996670 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.124078 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.124724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.125007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.125279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.126400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:49.127836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:49.129590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 114/213] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:49.135283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:49.157947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.158566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.158848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.159124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.159790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:49.160098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:49.161511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:49.170591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 100/214] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:49.176083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:49.213183 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:49.241151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b66e3d8-88a9-4aeb-892e-bdb17a736e30 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.275793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.276436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.276723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.277002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.277684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:49.279168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:49.279725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c461922-f35d-4b30-90b0-f8ce4de97cc3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.292938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:49.312462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.313141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.313447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.313723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] PATH_INFO: `/v3/users/936bb98925394e64b8d940e7167d9442` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.326846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] RBAC: Authorizing `identity:get_user(user_id=936bb98925394e64b8d940e7167d9442)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:49.328386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:49.329956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 101/215] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:49 2022] GET /identity/v3/users/936bb98925394e64b8d940e7167d9442 => generated 293 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:49.330739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:49.349355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 115/216] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:46:49 2022] POST /identity/v3/users => generated 293 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:46:49.349355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:49.444596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376336b3-7e28-427f-956f-2f681aedb314 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.483861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d661fa57-67cd-4331-94d3-602a88c37a84 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.487231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.487895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.488178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.488455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.502013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:49.508046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:49.508978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:49.509891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 102/217] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:49.515486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:49.520595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:49.527017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.527640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.527931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.528209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.528945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] RBAC: Adding query filter params (role.id=f1f68385aeb8421b80010aaf19d20d6e, user.id=936bb98925394e64b8d940e7167d9442, scope.system=all) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:49.529260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:49.530744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:49.536367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 116/218] 10.222.0.66 () {56 vars in 1308 bytes} [Fri Oct 28 19:46:49 2022] GET /identity/v3/role_assignments?user.id=936bb98925394e64b8d940e7167d9442&scope.system=all&role.id=f1f68385aeb8421b80010aaf19d20d6e => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:49.558099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:49.644476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a408c827-cd77-43fb-ab33-be3865d722e1 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.688519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.689435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.689902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.690180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.690869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:49.691181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:49.692623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:49.703632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 103/219] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:49 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:49.709576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:49.823015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e525f8fd-2b1d-4f70-8959-b302ad5b2a3b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:49.908039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:49.908687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:49.908970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:49.909247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] PATH_INFO: `/v3/users/6f018fc8d39842848a4271aaed35ae83` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:49.910455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] RBAC: Authorizing `identity:get_user(user_id=6f018fc8d39842848a4271aaed35ae83)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:49.911902 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:49.913880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 117/220] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:49 2022] GET /identity/v3/users/6f018fc8d39842848a4271aaed35ae83 => generated 297 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:49.919125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:50.025436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:50.040599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f3fe3-c082-43e2-88b4-7b3e9d211ac5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:50.075655 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:50.076272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:50.076551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:50.076825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:50.145699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:50.147167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:50.242405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:50.248694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 104/221] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:46:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:46:50.254464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:50.346053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:50.358506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730a7dfc-f156-4961-aba6-569c7f496ac4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:50.393734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b8592e-b526-4f3f-9251-2169eaaa7d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:50.394377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b8592e-b526-4f3f-9251-2169eaaa7d0f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:50.394659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b8592e-b526-4f3f-9251-2169eaaa7d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:50.394935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b8592e-b526-4f3f-9251-2169eaaa7d0f None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:50.396064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b8592e-b526-4f3f-9251-2169eaaa7d0f None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:50.397547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b8592e-b526-4f3f-9251-2169eaaa7d0f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:50.399841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 118/222] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:46:50 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:50.406113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:50.515627 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e7ba66-b8a1-433b-bd4b-d01cbe6b067f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:50.551752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb126df-a82b-445e-9547-8a3192ba4e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:50.552370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb126df-a82b-445e-9547-8a3192ba4e0f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:50.552648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb126df-a82b-445e-9547-8a3192ba4e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:50.552920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb126df-a82b-445e-9547-8a3192ba4e0f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:50.553678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb126df-a82b-445e-9547-8a3192ba4e0f None admin] RBAC: Adding query filter params (role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d, scope.project.id=c9cd84395d7f49969b6df473fd521e6d, user.id=6f018fc8d39842848a4271aaed35ae83) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:50.554546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb126df-a82b-445e-9547-8a3192ba4e0f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:50.556174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb126df-a82b-445e-9547-8a3192ba4e0f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:50.563015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 105/223] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:46:50 2022] GET /identity/v3/role_assignments?user.id=6f018fc8d39842848a4271aaed35ae83&scope.project.id=c9cd84395d7f49969b6df473fd521e6d&role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:51.730403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069e9f85-49e8-4100-96cd-759f9e2257f6 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:51.731133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069e9f85-49e8-4100-96cd-759f9e2257f6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:51.731133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069e9f85-49e8-4100-96cd-759f9e2257f6 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:51.732489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 119/224] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:51.738007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980528fd-9ea4-4821-bfdf-5f9577b94e8a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:51.738459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980528fd-9ea4-4821-bfdf-5f9577b94e8a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:51.738556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980528fd-9ea4-4821-bfdf-5f9577b94e8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:51.758351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-980528fd-9ea4-4821-bfdf-5f9577b94e8a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:51.759773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-980528fd-9ea4-4821-bfdf-5f9577b94e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:51.781590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a4c4a8-f19e-4386-acd1-40892a2ce479 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:51.781874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a4c4a8-f19e-4386-acd1-40892a2ce479 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:51.782157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a4c4a8-f19e-4386-acd1-40892a2ce479 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:51.783583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 120/225] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:51.789157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f8033-73df-4613-8794-524e5e04debb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:51.790884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f8033-73df-4613-8794-524e5e04debb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:51.790884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f8033-73df-4613-8794-524e5e04debb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:51.808655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3f5f8033-73df-4613-8794-524e5e04debb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:51.810134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3f5f8033-73df-4613-8794-524e5e04debb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:51.816498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:51.830739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 106/226] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:51.832081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc378ee-c186-4ab3-9502-0b05d7057d59 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:51.832368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc378ee-c186-4ab3-9502-0b05d7057d59 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:51.832635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc378ee-c186-4ab3-9502-0b05d7057d59 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:51.833900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 107/227] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:51.839748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8d307-c6aa-48a8-8786-3bcbaba359e1 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:51.840036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8d307-c6aa-48a8-8786-3bcbaba359e1 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:51.840316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c8d307-c6aa-48a8-8786-3bcbaba359e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:51.859182 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-33c8d307-c6aa-48a8-8786-3bcbaba359e1 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:51.860361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-33c8d307-c6aa-48a8-8786-3bcbaba359e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:51.909536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 121/228] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:51.932778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 108/229] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:52.103255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f8033-73df-4613-8794-524e5e04debb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.163749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:52.184570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8d307-c6aa-48a8-8786-3bcbaba359e1 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.205673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f8033-73df-4613-8794-524e5e04debb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.240601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.241240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.241588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.241875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.251798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:52.256720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:52.257984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 122/230] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:52.258785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.284874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c8d307-c6aa-48a8-8786-3bcbaba359e1 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.302430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:52.319954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.320580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.320867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.321144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.331317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:52.336824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:52.337745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:52.338517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 109/231] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:52.339154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.363006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cdb128-7bdd-4933-9fac-7a6210d95cf6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.382481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:52.396098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.396736 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.397018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.397302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.399155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:52.400603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:52.402210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 123/232] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:52.402984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.440493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1790112d-139f-4685-b096-83b32fb4956b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.474678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.475287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.475574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.476559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.477227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:52.477566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:52.478975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:52.487241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 110/233] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:52.487902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.509722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad1b438-9888-4af1-b884-94ce47715b9f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.528040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:52.544089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.544634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.544919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.545220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.545907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:52.546210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:52.547587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:52.555761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 124/234] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:52.556541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.593436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9196ef8-5e5c-4b30-8761-c3e2e44aa7c9 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.622634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:52.635812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.636421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.636700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.636976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] PATH_INFO: `/v3/groups` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.637628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:52.639099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:52.672209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d61224-3520-4a5a-a268-aa0ca504c4f4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.696710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 111/235] 10.222.0.66 () {60 vars in 1134 bytes} [Fri Oct 28 19:46:52 2022] POST /identity/v3/groups => generated 229 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:52.702402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.705522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.706178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.706466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.706765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] PATH_INFO: `/v3/users/936bb98925394e64b8d940e7167d9442` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.707984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] RBAC: Authorizing `identity:get_user(user_id=936bb98925394e64b8d940e7167d9442)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:52.709502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:52.716953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 125/236] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/users/936bb98925394e64b8d940e7167d9442 => generated 293 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:52.718372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.816048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:52.819991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b11aa-ba7d-4dee-83b0-2d4620f12c3d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.837939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9759f5de-b452-486f-b925-d4c371bfe286 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:52.857895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.858337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:52.858552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.858827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.859100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] PATH_INFO: `/v3/users/e67ec985b96346c6a84af6e33578bff6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.875649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] RBAC: Authorizing `identity:get_user(user_id=e67ec985b96346c6a84af6e33578bff6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:52.877191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:52.877423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bea709-5205-4522-99fe-f367f64f7bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:52.879986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bea709-5205-4522-99fe-f367f64f7bd6 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:52.879986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bea709-5205-4522-99fe-f367f64f7bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:52.879986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bea709-5205-4522-99fe-f367f64f7bd6 None admin] PATH_INFO: `/v3/system/users/936bb98925394e64b8d940e7167d9442/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:52.880218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 112/237] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/users/e67ec985b96346c6a84af6e33578bff6 => generated 293 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:52.880287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bea709-5205-4522-99fe-f367f64f7bd6 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=936bb98925394e64b8d940e7167d9442, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:52.883281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bea709-5205-4522-99fe-f367f64f7bd6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:52.886427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:52.971199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 126/238] 10.222.0.66 () {58 vars in 1334 bytes} [Fri Oct 28 19:46:52 2022] PUT /identity/v3/system/users/936bb98925394e64b8d940e7167d9442/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:46:52.987375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebad1404-80b9-4cfe-85ae-a12ab59ba5d5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:53.037826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ecb2fd-6ccf-453a-8613-c7a4496b7a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:53.039731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ecb2fd-6ccf-453a-8613-c7a4496b7a8a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:53.040034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ecb2fd-6ccf-453a-8613-c7a4496b7a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:53.040314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ecb2fd-6ccf-453a-8613-c7a4496b7a8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:53.041124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ecb2fd-6ccf-453a-8613-c7a4496b7a8a None admin] RBAC: Adding query filter params (role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d, user.id=e67ec985b96346c6a84af6e33578bff6, scope.system=all) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:53.041467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ecb2fd-6ccf-453a-8613-c7a4496b7a8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:53.042964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ecb2fd-6ccf-453a-8613-c7a4496b7a8a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:53.049108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 113/239] 10.222.0.66 () {56 vars in 1308 bytes} [Fri Oct 28 19:46:52 2022] GET /identity/v3/role_assignments?user.id=e67ec985b96346c6a84af6e33578bff6&scope.system=all&role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d => generated 226 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:54.970395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85ce36-779d-46fd-9185-4b2792079684 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:54.970940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85ce36-779d-46fd-9185-4b2792079684 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:54.970940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85ce36-779d-46fd-9185-4b2792079684 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:54.972485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 127/240] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:54.978329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e685285-788c-4acb-ad9c-f426b0b0f286 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:54.978751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e685285-788c-4acb-ad9c-f426b0b0f286 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:54.978943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e685285-788c-4acb-ad9c-f426b0b0f286 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:54.997100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6e685285-788c-4acb-ad9c-f426b0b0f286 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:54.998336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6e685285-788c-4acb-ad9c-f426b0b0f286 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:55.018644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:55.085370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 114/241] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:55.158886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f69fb-d5fc-4cf2-a11f-89aaaf481c5e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.159170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f69fb-d5fc-4cf2-a11f-89aaaf481c5e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.159623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f69fb-d5fc-4cf2-a11f-89aaaf481c5e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.160986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 128/242] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:55.166920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0704eb3-60e0-4f30-bd01-c72b7453632c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.167213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0704eb3-60e0-4f30-bd01-c72b7453632c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.167516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0704eb3-60e0-4f30-bd01-c72b7453632c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.185928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ce0f54-e82e-4950-96cf-eb8f5bcc86fe None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.186212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ce0f54-e82e-4950-96cf-eb8f5bcc86fe None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.186587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b0704eb3-60e0-4f30-bd01-c72b7453632c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:55.186691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ce0f54-e82e-4950-96cf-eb8f5bcc86fe None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.187761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b0704eb3-60e0-4f30-bd01-c72b7453632c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:55.188226 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 129/243] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:55.194239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dd022-652a-4c93-8e97-754cdcf8ee8e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.194571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dd022-652a-4c93-8e97-754cdcf8ee8e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.195042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dd022-652a-4c93-8e97-754cdcf8ee8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.214686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3e9dd022-652a-4c93-8e97-754cdcf8ee8e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:55.215886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3e9dd022-652a-4c93-8e97-754cdcf8ee8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:55.261203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 115/244] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:55.284974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 130/245] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:46:55.352073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0704eb3-60e0-4f30-bd01-c72b7453632c None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:55.433492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:55.452351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0704eb3-60e0-4f30-bd01-c72b7453632c None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:55.486599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:55.487237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.487515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.487787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.488874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:55.490336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:55.492689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 116/246] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:46:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:55.498243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dd022-652a-4c93-8e97-754cdcf8ee8e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:55.561753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:55.598319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dd022-652a-4c93-8e97-754cdcf8ee8e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:55.611625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:55.632156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:55.632857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.633142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.633710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] PATH_INFO: `/v3/groups` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.635112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:55.637169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:55.664925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d8b97f-cc92-43f1-8a79-c0366b3049db None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:55.699922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:55.700525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.700801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.701069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.711437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:55.716439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:55.717377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:55.718239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 117/247] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:55.718915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:55.729375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 131/248] 10.222.0.66 () {60 vars in 1134 bytes} [Fri Oct 28 19:46:55 2022] POST /identity/v3/groups => generated 226 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:46:55.729375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:55.731774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:55.848283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c326db54-6d31-4ce3-b977-f732bf83f54f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:55.858639 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed65871-e13e-4112-ab27-9b194216a3e7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:55.889649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:55.890303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.890584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.890891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.896580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:55.897208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:55.897577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:55.898516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:55.898717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:55.899025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:55.900429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:55.904860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:55.909047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 132/249] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:55.910184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:55.911037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:55.912372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 118/250] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:55.915450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:55.917588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:55.956193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:56.017339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:56.055934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fcc6f1-3972-4d5a-aba1-cce5b5f0373f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:56.064938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe81b1d-3d62-4222-9eb1-d23d51c9e34d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:56.106860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:56.107758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:56.107914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:56.108334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:56.108577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:56.108867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:56.109121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:56.109225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] PATH_INFO: `/v3/users/e67ec985b96346c6a84af6e33578bff6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:56.110151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:56.110574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] RBAC: Authorizing `identity:get_user(user_id=e67ec985b96346c6a84af6e33578bff6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:56.110778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:56.112237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:56.112532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:56.117378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 133/251] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:55 2022] GET /identity/v3/users/e67ec985b96346c6a84af6e33578bff6 => generated 293 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:56.121679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:56.128698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 119/252] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:46:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:56.134796 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:56.267386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd7843-06ed-4438-aeb4-6939ad0f6f80 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:56.267576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea20a4-dd8f-4e52-bc74-3b68f60ac3ee None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:56.269972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:56.305481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:56.312668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:56.313556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:56.313844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:56.313947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5f31d1-e924-4c5c-b68a-3445a8837c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:56.314127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] PATH_INFO: `/v3/users/936bb98925394e64b8d940e7167d9442` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:56.314582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f31d1-e924-4c5c-b68a-3445a8837c80 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:56.314951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f31d1-e924-4c5c-b68a-3445a8837c80 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:56.315232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f31d1-e924-4c5c-b68a-3445a8837c80 None admin] PATH_INFO: `/v3/system/users/e67ec985b96346c6a84af6e33578bff6/roles/e7600b54d8dc4d2d9a7c5895b14e1f6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:56.315332 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] RBAC: Authorizing `identity:get_user(user_id=936bb98925394e64b8d940e7167d9442)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:56.316733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5f31d1-e924-4c5c-b68a-3445a8837c80 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=e67ec985b96346c6a84af6e33578bff6, role_id=e7600b54d8dc4d2d9a7c5895b14e1f6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:56.316897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:56.318123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5f31d1-e924-4c5c-b68a-3445a8837c80 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:56.318525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 120/253] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:46:56 2022] GET /identity/v3/users/936bb98925394e64b8d940e7167d9442 => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:56.324799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:56.383867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 134/254] 10.222.0.66 () {58 vars in 1334 bytes} [Fri Oct 28 19:46:56 2022] PUT /identity/v3/system/users/e67ec985b96346c6a84af6e33578bff6/roles/e7600b54d8dc4d2d9a7c5895b14e1f6d => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:46:56.454061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe8c2f-4202-43c9-ac89-bcf0502ec976 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:56.500431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151a1212-b366-45a5-985d-c227566b2c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:56.501399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151a1212-b366-45a5-985d-c227566b2c0d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:56.501855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151a1212-b366-45a5-985d-c227566b2c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:56.502326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151a1212-b366-45a5-985d-c227566b2c0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:56.503237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151a1212-b366-45a5-985d-c227566b2c0d None admin] RBAC: Adding query filter params (role.id=f1f68385aeb8421b80010aaf19d20d6e, user.id=936bb98925394e64b8d940e7167d9442, scope.system=all) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:56.503722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151a1212-b366-45a5-985d-c227566b2c0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:56.505568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151a1212-b366-45a5-985d-c227566b2c0d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:56.520202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 121/255] 10.222.0.66 () {56 vars in 1308 bytes} [Fri Oct 28 19:46:56 2022] GET /identity/v3/role_assignments?user.id=936bb98925394e64b8d940e7167d9442&scope.system=all&role.id=f1f68385aeb8421b80010aaf19d20d6e => generated 510 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:58.172403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e0171-9e0f-474d-b374-f62d3cfa9872 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.172905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e0171-9e0f-474d-b374-f62d3cfa9872 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.173003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e0171-9e0f-474d-b374-f62d3cfa9872 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.174460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 135/256] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:58.181047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562f82ca-44b7-4c59-8b51-1c98d893442f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.181509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562f82ca-44b7-4c59-8b51-1c98d893442f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.181710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562f82ca-44b7-4c59-8b51-1c98d893442f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.210966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-562f82ca-44b7-4c59-8b51-1c98d893442f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:58.212379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-562f82ca-44b7-4c59-8b51-1c98d893442f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:58.291595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 122/257] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:58 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 28 19:46:58.602418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd57d42f-795d-4105-a6fb-cbefef15b65f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.602763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd57d42f-795d-4105-a6fb-cbefef15b65f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.602959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd57d42f-795d-4105-a6fb-cbefef15b65f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.604309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 136/258] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:58.610155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34317265-a073-4fe0-8c33-b36e377d5f23 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.610743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34317265-a073-4fe0-8c33-b36e377d5f23 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.611214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34317265-a073-4fe0-8c33-b36e377d5f23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.626662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:46:58.639011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-34317265-a073-4fe0-8c33-b36e377d5f23 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:58.640571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-34317265-a073-4fe0-8c33-b36e377d5f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:58.649560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 123/259] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:58 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 28 19:46:58.825204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd57d42f-795d-4105-a6fb-cbefef15b65f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:58.840883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd57d42f-795d-4105-a6fb-cbefef15b65f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:58.848742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:58.849420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.849706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.849983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.865743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:58.871841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:58.872772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:58.873595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 137/260] 10.222.0.66 () {56 vars in 1120 bytes} [Fri Oct 28 19:46:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 48 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:46:58.880338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34317265-a073-4fe0-8c33-b36e377d5f23 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:58.894523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34317265-a073-4fe0-8c33-b36e377d5f23 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:58.902767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:58.903647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.904145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.904614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.905509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:58.906009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:58.908002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 19:46:58.908002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 19:46:58.908002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 19:46:58.909285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d72d30-d769-405f-a98a-97bd1d6e47ff admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:58.921874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 124/261] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:46:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:58.927247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:58.936110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13428a7e-44ac-4b88-b2a2-68655e94213b None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.936402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13428a7e-44ac-4b88-b2a2-68655e94213b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.936683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13428a7e-44ac-4b88-b2a2-68655e94213b None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.938221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 125/262] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:46:58.943279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b190d-7567-4b94-8d5a-d2c449bc1637 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:58.949231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c8dfe0-99ce-45f9-9af1-99ece539fb4b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.949571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c8dfe0-99ce-45f9-9af1-99ece539fb4b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.949851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c8dfe0-99ce-45f9-9af1-99ece539fb4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.961138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:58.961821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:58.962101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:58.962375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:58.974970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:58.977547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 19:46:58.977547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 19:46:58.977547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 19:46:58.978803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:58.980472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 138/263] 10.222.0.66 () {56 vars in 1201 bytes} [Fri Oct 28 19:46:58 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:58.984928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-79c8dfe0-99ce-45f9-9af1-99ece539fb4b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:58.985984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-79c8dfe0-99ce-45f9-9af1-99ece539fb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:58.987626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.003406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af17acae-73a9-428b-8e97-44ff86346444 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.005999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 126/264] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:58 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 384 bytes (3 switches on core 0) Oct 28 19:46:59.020385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.021045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.021360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.021633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.022875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.024949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.031752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 139/265] 10.222.0.66 () {56 vars in 1207 bytes} [Fri Oct 28 19:46:58 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:59.040178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c8dfe0-99ce-45f9-9af1-99ece539fb4b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.050237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c8dfe0-99ce-45f9-9af1-99ece539fb4b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.055614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31ef9a-c0d1-4287-a1ee-2750ce9706dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.056330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31ef9a-c0d1-4287-a1ee-2750ce9706dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.056657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31ef9a-c0d1-4287-a1ee-2750ce9706dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.056936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31ef9a-c0d1-4287-a1ee-2750ce9706dc admin admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/groups/9dc827da7ca448df96b04026ba26ab9a/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.061259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31ef9a-c0d1-4287-a1ee-2750ce9706dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d04c8e059934c64b4f015cf4a0e6641, group_id=9dc827da7ca448df96b04026ba26ab9a, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.062412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-9d31ef9a-c0d1-4287-a1ee-2750ce9706dc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 19:46:59.062412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 19:46:59.062412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 19:46:59.064448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31ef9a-c0d1-4287-a1ee-2750ce9706dc admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.137958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 127/266] 10.222.0.66 () {58 vars in 1463 bytes} [Fri Oct 28 19:46:59 2022] PUT /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/groups/9dc827da7ca448df96b04026ba26ab9a/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:59.204253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.216528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d7febf-4e20-4541-ad7c-c08b58a063f9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.297435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.298097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.298386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.298666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.315624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:59.325289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:59.326270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:46:59.327086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 140/267] 10.222.0.66 () {56 vars in 1135 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 123 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:59.334625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5a2f1-155c-41c4-a5d4-ad6a6fecd28b None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.335173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5a2f1-155c-41c4-a5d4-ad6a6fecd28b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.335454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5a2f1-155c-41c4-a5d4-ad6a6fecd28b None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.336843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 128/268] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:46:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:46:59.337884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.342902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9cd4c-5c2d-4e3e-9d0b-eccd6d1853f5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.343191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9cd4c-5c2d-4e3e-9d0b-eccd6d1853f5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.343463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9cd4c-5c2d-4e3e-9d0b-eccd6d1853f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.371351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abe547f-e706-4ac6-a0c1-2a0924a12ab3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.382532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-faa9cd4c-5c2d-4e3e-9d0b-eccd6d1853f5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:46:59.384381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-faa9cd4c-5c2d-4e3e-9d0b-eccd6d1853f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:46:59.388703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.389397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.389637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.389964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.390665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:59.390973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.396797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 19:46:59.396797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 19:46:59.396797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 19:46:59.400021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.421504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 141/269] 10.222.0.66 () {56 vars in 1132 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:59.424767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.442425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20fec09-ae2b-4e7f-b3e4-e6fab80a2207 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.458574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.459608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.460675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.461152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.462787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.465020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.467190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 142/270] 10.222.0.66 () {56 vars in 1201 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:59.474278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.488372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff0794-eaf5-4e68-8f91-b899965d1656 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.500879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.501896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.502381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.502856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.504915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.508597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.510699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 143/271] 10.222.0.66 () {56 vars in 1207 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:59.516312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.528409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 129/272] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:46:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:46:59.532768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa9cd4c-5c2d-4e3e-9d0b-eccd6d1853f5 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.537519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f925398-c8fc-447d-b33d-095473cf008d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.550124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.551095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.551651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.552141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/groups/9dc827da7ca448df96b04026ba26ab9a/roles/335ad63ce3214c9aa7e4f4f058f97465` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.554287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa9cd4c-5c2d-4e3e-9d0b-eccd6d1853f5 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.556155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d04c8e059934c64b4f015cf4a0e6641, group_id=9dc827da7ca448df96b04026ba26ab9a, role_id=335ad63ce3214c9aa7e4f4f058f97465)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.558066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 19:46:59.558066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 19:46:59.558066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 19:46:59.561732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.566843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.567743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.568219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.568687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.588221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:46:59.593473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:46:59.594364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:46:59.595150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 130/273] 10.222.0.66 () {56 vars in 1120 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 63 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:46:59.600792 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.615900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2009268-4eef-42a3-9c32-e3f239e5e4b6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.624877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.625511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.625886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.626080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.626751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:46:59.627051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.628725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.643303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 131/274] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:46:59.649971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.658288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 144/275] 10.222.0.66 () {58 vars in 1463 bytes} [Fri Oct 28 19:46:59 2022] PUT /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641/groups/9dc827da7ca448df96b04026ba26ab9a/roles/335ad63ce3214c9aa7e4f4f058f97465 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:46:59.668556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d01f01-5986-4927-bd61-2485f6943155 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.743106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.743755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.744035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.744305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.745426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.746457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 19:46:59.746457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 19:46:59.746457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 19:46:59.747657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.750217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 132/276] 10.222.0.66 () {56 vars in 1201 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:59.754748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.770446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5382259-e934-400d-8a69-4c608064c652 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.786337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.787345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.793502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.793502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.793502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.794503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.796538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 145/277] 10.222.0.66 () {56 vars in 1207 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:46:59.802444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.813286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe69a7-6c06-485d-a46f-a7c21e092724 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:46:59.824873 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:46:59.825575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:46:59.825856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:46:59.826129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d/groups/9dc827da7ca448df96b04026ba26ab9a/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:46:59.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9cd84395d7f49969b6df473fd521e6d, group_id=9dc827da7ca448df96b04026ba26ab9a, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:46:59.830711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:46:59.864546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:46:59.939797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:00.003863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119a901-f7ee-40e0-a094-ddf2d2701862 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:00.016470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 133/278] 10.222.0.66 () {58 vars in 1463 bytes} [Fri Oct 28 19:46:59 2022] PUT /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d/groups/9dc827da7ca448df96b04026ba26ab9a/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:47:00.050443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:00.051174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:00.051418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:00.051689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:00.063955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:00.069891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:00.070974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:00.071762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 146/279] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:46:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:00.077429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:00.099582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:00.218409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f56c0-993f-4749-895c-1640dd63b9a0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:00.260837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:00.261518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:00.261801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:00.262074 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:00.262758 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:00.263058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:00.264490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:00.274445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 134/280] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:47:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:00.280210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:00.418410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c7d534-3ed6-459f-9744-4e4260abc11e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:00.431951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:00.464890 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2b71a9-f119-418a-9367-3677d2cf8139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:00.465583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b71a9-f119-418a-9367-3677d2cf8139 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:00.465868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b71a9-f119-418a-9367-3677d2cf8139 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:00.466142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b71a9-f119-418a-9367-3677d2cf8139 None admin] PATH_INFO: `/v3/users/e67ec985b96346c6a84af6e33578bff6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:00.467291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2b71a9-f119-418a-9367-3677d2cf8139 None admin] RBAC: Authorizing `identity:get_user(user_id=e67ec985b96346c6a84af6e33578bff6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:00.468809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2b71a9-f119-418a-9367-3677d2cf8139 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:00.470662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 147/281] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:47:00 2022] GET /identity/v3/users/e67ec985b96346c6a84af6e33578bff6 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:00.477158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:00.580211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:00.606020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f7416f-93b9-4572-b8e4-ee05d1e978d5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:00.645931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4772df00-6eac-48df-891e-c2404f7e8e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:00.646550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4772df00-6eac-48df-891e-c2404f7e8e9f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:00.646830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4772df00-6eac-48df-891e-c2404f7e8e9f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:00.647103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4772df00-6eac-48df-891e-c2404f7e8e9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:00.647985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4772df00-6eac-48df-891e-c2404f7e8e9f None admin] RBAC: Adding query filter params (role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d, user.id=e67ec985b96346c6a84af6e33578bff6, scope.system=all) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:00.648295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4772df00-6eac-48df-891e-c2404f7e8e9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:00.649826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4772df00-6eac-48df-891e-c2404f7e8e9f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:00.666664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 135/282] 10.222.0.66 () {56 vars in 1308 bytes} [Fri Oct 28 19:47:00 2022] GET /identity/v3/role_assignments?user.id=e67ec985b96346c6a84af6e33578bff6&scope.system=all&role.id=e7600b54d8dc4d2d9a7c5895b14e1f6d => generated 510 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:01.057089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198f496-ef3f-4448-9d4b-2bef0f80769c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.057773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198f496-ef3f-4448-9d4b-2bef0f80769c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.058248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198f496-ef3f-4448-9d4b-2bef0f80769c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.059794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 148/283] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:01.065426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9151cfa0-632e-4ce3-82c8-819fcb4e1d9a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.065972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9151cfa0-632e-4ce3-82c8-819fcb4e1d9a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.066453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9151cfa0-632e-4ce3-82c8-819fcb4e1d9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.090926 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-9151cfa0-632e-4ce3-82c8-819fcb4e1d9a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:01.092456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9151cfa0-632e-4ce3-82c8-819fcb4e1d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:01.165046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 136/284] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:01 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 28 19:47:01.517576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e314fcc-38cd-4879-9555-5f29d4bdbf5b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.518114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e314fcc-38cd-4879-9555-5f29d4bdbf5b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.518114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e314fcc-38cd-4879-9555-5f29d4bdbf5b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.519733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 149/285] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:01.526785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6f276-731f-4693-b80c-da6f9d1ff4e4 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.527220 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6f276-731f-4693-b80c-da6f9d1ff4e4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.527399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6f276-731f-4693-b80c-da6f9d1ff4e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.558292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-89c6f276-731f-4693-b80c-da6f9d1ff4e4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:01.559507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-89c6f276-731f-4693-b80c-da6f9d1ff4e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:01.567997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 137/286] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:01 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 28 19:47:01.698811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e314fcc-38cd-4879-9555-5f29d4bdbf5b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:01.714717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e314fcc-38cd-4879-9555-5f29d4bdbf5b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:01.719564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:01.720213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.720496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.720770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.736857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:01.742545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:01.743413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:01.744201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 150/287] 10.222.0.66 () {56 vars in 1135 bytes} [Fri Oct 28 19:47:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 46 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:01.749628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6f276-731f-4693-b80c-da6f9d1ff4e4 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:01.763395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6f276-731f-4693-b80c-da6f9d1ff4e4 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:01.774683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:01.775254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.775542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.775821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.776503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:01.776812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:01.778466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:01.795311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 138/288] 10.222.0.66 () {56 vars in 1132 bytes} [Fri Oct 28 19:47:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:01.801266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:01.814860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e931a284-9259-4532-b2bb-bd215fba190b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:01.824015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:01.824720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.825011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.825288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.826465 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:01.828437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:01.830194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 151/289] 10.222.0.66 () {56 vars in 1201 bytes} [Fri Oct 28 19:47:01 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:01.837041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:01.855053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4487d92d-09c5-490f-9475-f6d4e14535d8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:01.863278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:01.863960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.864250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.864528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.865794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:01.869102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:01.871094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 139/290] 10.222.0.66 () {56 vars in 1207 bytes} [Fri Oct 28 19:47:01 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:01.876577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:01.891804 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a8d677-3559-47bb-a6b7-aa303cf4797c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:01.903442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:01.904179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.904468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.904743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d/groups/9dc827da7ca448df96b04026ba26ab9a/roles/335ad63ce3214c9aa7e4f4f058f97465` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.907675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9cd84395d7f49969b6df473fd521e6d, group_id=9dc827da7ca448df96b04026ba26ab9a, role_id=335ad63ce3214c9aa7e4f4f058f97465)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:01.910411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:01.963246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:01.973682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67308c61-e4c7-484c-b5b1-e72c7b70bfee admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:01.978841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:01.979468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:01.979747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:01.980020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:01.991535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:01.996561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 152/291] 10.222.0.66 () {58 vars in 1463 bytes} [Fri Oct 28 19:47:01 2022] PUT /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d/groups/9dc827da7ca448df96b04026ba26ab9a/roles/335ad63ce3214c9aa7e4f4f058f97465 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:02.000137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:02.002299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:02.004283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 140/292] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:47:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 39 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:02.022119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:02.114859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2c0ff-b86b-4ed4-85e2-29bd34495e02 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:02.121268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:02.121936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.122225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.122502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.123184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:02.123489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:02.125123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:02.137573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 153/293] 10.222.0.66 () {56 vars in 1120 bytes} [Fri Oct 28 19:47:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:02.142973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:02.163505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da2c7f-b4b7-428e-8157-3ba8d3d5c38c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:02.171034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:02.171696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.171976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.172251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] PATH_INFO: `/v3/groups/a2da0988336e48ecaa5182ca4e870080` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.183454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] RBAC: Authorizing `identity:get_group(group_id=a2da0988336e48ecaa5182ca4e870080)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:02.185741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:02.187425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 141/294] 10.222.0.66 () {56 vars in 1201 bytes} [Fri Oct 28 19:47:02 2022] GET /identity/v3/groups/a2da0988336e48ecaa5182ca4e870080 => generated 229 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:02.193146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:02.205528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4303de89-70dd-4512-b3ed-91b6167f815c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:02.214219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb1c56b-1cc8-4565-b8af-74bbf070d150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:02.215888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb1c56b-1cc8-4565-b8af-74bbf070d150 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.215888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb1c56b-1cc8-4565-b8af-74bbf070d150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.215888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb1c56b-1cc8-4565-b8af-74bbf070d150 admin admin] PATH_INFO: `/v3/projects/9255f3850e094deb824d58d03e377387` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.219647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb1c56b-1cc8-4565-b8af-74bbf070d150 admin admin] RBAC: Authorizing `identity:get_project(project_id=9255f3850e094deb824d58d03e377387)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:02.220409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb1c56b-1cc8-4565-b8af-74bbf070d150 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:02.225348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 154/295] 10.222.0.66 () {56 vars in 1207 bytes} [Fri Oct 28 19:47:02 2022] GET /identity/v3/projects/9255f3850e094deb824d58d03e377387 => generated 344 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:02.228917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:02.244370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ad8766-7c29-43ac-a57b-003171c0daf8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:02.251592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad04b1-ac9b-4da2-9c98-46a8226044f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:02.252284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad04b1-ac9b-4da2-9c98-46a8226044f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.252568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad04b1-ac9b-4da2-9c98-46a8226044f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.252839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad04b1-ac9b-4da2-9c98-46a8226044f1 admin admin] PATH_INFO: `/v3/projects/9255f3850e094deb824d58d03e377387/groups/a2da0988336e48ecaa5182ca4e870080/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.254779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cad04b1-ac9b-4da2-9c98-46a8226044f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9255f3850e094deb824d58d03e377387, group_id=a2da0988336e48ecaa5182ca4e870080, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:02.261985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cad04b1-ac9b-4da2-9c98-46a8226044f1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:02.343864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 142/296] 10.222.0.66 () {58 vars in 1463 bytes} [Fri Oct 28 19:47:02 2022] PUT /identity/v3/projects/9255f3850e094deb824d58d03e377387/groups/a2da0988336e48ecaa5182ca4e870080/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:02.615152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa526bd-fe82-4f31-90d0-e92286863e67 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.615152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa526bd-fe82-4f31-90d0-e92286863e67 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.615152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa526bd-fe82-4f31-90d0-e92286863e67 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.615152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 155/297] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:02.621595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f12c058-99b5-4294-aacf-c508f09f964f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.622128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f12c058-99b5-4294-aacf-c508f09f964f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.622128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f12c058-99b5-4294-aacf-c508f09f964f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.644574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8f12c058-99b5-4294-aacf-c508f09f964f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:02.645812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8f12c058-99b5-4294-aacf-c508f09f964f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:02.675432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:02.738966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 143/298] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:02.906739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f029ad-f537-45d0-9f17-350a79247a9c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.907053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f029ad-f537-45d0-9f17-350a79247a9c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.907210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f029ad-f537-45d0-9f17-350a79247a9c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.908806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 156/299] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:02.914381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e222f69-3d83-4b34-a30c-e405f2fa3818 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:02.914677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e222f69-3d83-4b34-a30c-e405f2fa3818 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:02.914962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e222f69-3d83-4b34-a30c-e405f2fa3818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:02.933834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2e222f69-3d83-4b34-a30c-e405f2fa3818 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:02.935408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2e222f69-3d83-4b34-a30c-e405f2fa3818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:03.009509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 144/300] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:47:03.066384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f029ad-f537-45d0-9f17-350a79247a9c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:03.175351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:03.195607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f029ad-f537-45d0-9f17-350a79247a9c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:03.240748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d914be8-5bba-48df-a1e1-6813a7b99a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:03.241465 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d914be8-5bba-48df-a1e1-6813a7b99a78 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:03.241751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d914be8-5bba-48df-a1e1-6813a7b99a78 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:03.242028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d914be8-5bba-48df-a1e1-6813a7b99a78 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:03.252782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d914be8-5bba-48df-a1e1-6813a7b99a78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3d914be8-5bba-48df-a1e1-6813a7b99a78 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:03.261588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:03.262498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:03.263680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 157/301] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:03.270584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e222f69-3d83-4b34-a30c-e405f2fa3818 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:03.344472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4be109-7482-4faf-be2a-e8dcf6f65891 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:03.344870 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4be109-7482-4faf-be2a-e8dcf6f65891 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:03.345155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4be109-7482-4faf-be2a-e8dcf6f65891 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:03.346612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 158/302] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:47:03.351977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818ea3ea-5e34-4e3e-9e99-48bc4803fac9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:03.352267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818ea3ea-5e34-4e3e-9e99-48bc4803fac9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:03.352541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818ea3ea-5e34-4e3e-9e99-48bc4803fac9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:03.374308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-818ea3ea-5e34-4e3e-9e99-48bc4803fac9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:03.375771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-818ea3ea-5e34-4e3e-9e99-48bc4803fac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:03.388668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:03.403100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e222f69-3d83-4b34-a30c-e405f2fa3818 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:03.452724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:03.453969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:03.454844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:03.455299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:03.456418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:03.457155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:03.459489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:03.465610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 159/303] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:47:03.471520 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 145/304] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:47:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:03.476451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818ea3ea-5e34-4e3e-9e99-48bc4803fac9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:03.479284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:03.603093 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9badd645-bb81-40ae-9607-2fd3b3d9f55c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:03.646750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:03.648414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:03.648911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:03.649407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:03.664409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:03.675049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:03.719617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818ea3ea-5e34-4e3e-9e99-48bc4803fac9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:03.740003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:03.743275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:03.793128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:03.793811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:03.799306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:03.799306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:03.815567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:03.821837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:03.824462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:03.825294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 160/305] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:03.836786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:03.885607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 146/306] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:47:03.887404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:03.954164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:03.976587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96558e68-efb3-4010-a536-1f701800f20b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.028073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.030608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.030608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.030761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e863cc2-2970-43c7-abe7-a70cb0417ead None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.030836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.044062 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:04.054355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:04.061193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:04.062692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 161/307] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:04.063646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.078532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.080381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.080838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.081269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.082749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:04.083617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:04.086058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:04.098372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 147/308] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:04.099532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.211111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6728ee-b252-491c-aab7-d69380393c78 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.254254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90d3849-bfee-4123-827e-aba722cc82d3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.254825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.255709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.256193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.257206 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.259074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:04.260802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:04.262717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 162/309] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:04.266028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.272573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:04.305147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.305819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.306105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.306382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.307358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] RBAC: Adding query filter params (name=member) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:04.307682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:04.310335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:04.322848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 148/310] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:04.325987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.346744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:04.409044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0654c3b7-1a88-4d68-a252-386f1cd696d5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.453099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.453769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.454058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.454339 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.463792 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8577395e-23ba-4800-9bf9-c964d405a878 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.510516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.511081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.511367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.511644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.512823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:04.514447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:04.516325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 149/311] 10.222.0.66 () {56 vars in 1180 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:04.517219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.550903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:04.554320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:04.568408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:04.637949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6427915-da4f-49eb-bd13-ad48070c406b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.642439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:04.686184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.686839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.687097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.687375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.703605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 163/312] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:47:04.705122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.779413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:04.780941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:04.837391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:04.859378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c8567c-3059-4155-b3f8-85ebf986cd8a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:04.900762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4f6114-3526-439b-9b5d-613fcf0f0044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:04.901925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4f6114-3526-439b-9b5d-613fcf0f0044 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.902506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4f6114-3526-439b-9b5d-613fcf0f0044 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.902932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4f6114-3526-439b-9b5d-613fcf0f0044 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.904361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4f6114-3526-439b-9b5d-613fcf0f0044 None admin] RBAC: Adding query filter params (group.id=9dc827da7ca448df96b04026ba26ab9a, role.id=f1f68385aeb8421b80010aaf19d20d6e, scope.project.id=4d04c8e059934c64b4f015cf4a0e6641) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:04.904858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4f6114-3526-439b-9b5d-613fcf0f0044 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:04.907154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4f6114-3526-439b-9b5d-613fcf0f0044 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:04.910278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 150/313] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:47:04.911060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.916552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 164/314] 10.222.0.66 () {56 vars in 1376 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/role_assignments?group.id=9dc827da7ca448df96b04026ba26ab9a&scope.project.id=4d04c8e059934c64b4f015cf4a0e6641&role.id=f1f68385aeb8421b80010aaf19d20d6e => generated 611 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:04.918109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888c899-0a67-40d2-84d7-244bf140821c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:04.918398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888c899-0a67-40d2-84d7-244bf140821c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:04.918678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888c899-0a67-40d2-84d7-244bf140821c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:04.919994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 165/315] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:04.920726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e888c899-0a67-40d2-84d7-244bf140821c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:04.964679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:05.076698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2973af3d-806b-4f95-a0d0-758ef5765a61 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.079810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e888c899-0a67-40d2-84d7-244bf140821c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.124899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012bcb71-bee1-41f6-b417-369972425ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.125984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012bcb71-bee1-41f6-b417-369972425ff7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.126470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012bcb71-bee1-41f6-b417-369972425ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.126949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012bcb71-bee1-41f6-b417-369972425ff7 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.129209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012bcb71-bee1-41f6-b417-369972425ff7 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:05.129883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.130539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.130827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.131110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] PATH_INFO: `/v3/projects/4d04c8e059934c64b4f015cf4a0e6641` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.132472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012bcb71-bee1-41f6-b417-369972425ff7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:05.132857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] RBAC: Authorizing `identity:get_project(project_id=4d04c8e059934c64b4f015cf4a0e6641)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:05.134375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:05.134884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 166/316] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:05.136258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea96824d-704b-448f-a12e-1786bd8532fb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.136740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 151/317] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:47:04 2022] GET /identity/v3/projects/4d04c8e059934c64b4f015cf4a0e6641 => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:05.136846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea96824d-704b-448f-a12e-1786bd8532fb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.137299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea96824d-704b-448f-a12e-1786bd8532fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.142092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:05.158333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:05.160665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ea96824d-704b-448f-a12e-1786bd8532fb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:05.161934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ea96824d-704b-448f-a12e-1786bd8532fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:05.247382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 167/318] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:05.248243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea96824d-704b-448f-a12e-1786bd8532fb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:05.279331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4543a7-d152-4e5f-bc7b-38d8ab59ea4d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.279795 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:05.330740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95389ad-6664-4cf6-93b7-875f4307fb9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.331336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95389ad-6664-4cf6-93b7-875f4307fb9d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.331619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95389ad-6664-4cf6-93b7-875f4307fb9d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.331892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95389ad-6664-4cf6-93b7-875f4307fb9d None admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.333018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95389ad-6664-4cf6-93b7-875f4307fb9d None admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:05.334508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95389ad-6664-4cf6-93b7-875f4307fb9d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:05.337101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 152/319] 10.222.0.66 () {56 vars in 1180 bytes} [Fri Oct 28 19:47:05 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:05.338300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5792b90a-7dc6-4d78-8ac5-5d2e52cd3ca9 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.338586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5792b90a-7dc6-4d78-8ac5-5d2e52cd3ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.338864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5792b90a-7dc6-4d78-8ac5-5d2e52cd3ca9 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.340084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 153/320] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:05 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:05.343699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5792b90a-7dc6-4d78-8ac5-5d2e52cd3ca9 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:05.384018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea96824d-704b-448f-a12e-1786bd8532fb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.421064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e264af-2650-40b5-8195-710ba812639e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.421860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e264af-2650-40b5-8195-710ba812639e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.422148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e264af-2650-40b5-8195-710ba812639e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.422430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e264af-2650-40b5-8195-710ba812639e None admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.423609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e264af-2650-40b5-8195-710ba812639e None admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:05.425349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e264af-2650-40b5-8195-710ba812639e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:05.426976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 168/321] 10.222.0.66 () {56 vars in 1180 bytes} [Fri Oct 28 19:47:05 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:05.428118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e03d9c-7909-40be-a22a-8399d1fbc126 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.428405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e03d9c-7909-40be-a22a-8399d1fbc126 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.429222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e03d9c-7909-40be-a22a-8399d1fbc126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.435511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:05.457607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-14e03d9c-7909-40be-a22a-8399d1fbc126 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:05.458805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-14e03d9c-7909-40be-a22a-8399d1fbc126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:05.464573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5792b90a-7dc6-4d78-8ac5-5d2e52cd3ca9 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.516317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.516935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.517221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.517528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.518264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] RBAC: Adding query filter params (group.id=9dc827da7ca448df96b04026ba26ab9a, role.id=335ad63ce3214c9aa7e4f4f058f97465, scope.project.id=4d04c8e059934c64b4f015cf4a0e6641) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:05.518576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:05.520015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:05.528478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 154/322] 10.222.0.66 () {56 vars in 1376 bytes} [Fri Oct 28 19:47:05 2022] GET /identity/v3/role_assignments?group.id=9dc827da7ca448df96b04026ba26ab9a&scope.project.id=4d04c8e059934c64b4f015cf4a0e6641&role.id=335ad63ce3214c9aa7e4f4f058f97465 => generated 611 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:05.529350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:05.548702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 169/323] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:05.549633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e03d9c-7909-40be-a22a-8399d1fbc126 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:05.586366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:05.664148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454c3dbb-9094-4bbf-b9b6-29203dd6adae None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.713397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e03d9c-7909-40be-a22a-8399d1fbc126 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.719125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.719757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.720037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.720311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.721048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] RBAC: Adding query filter params (group.id=9dc827da7ca448df96b04026ba26ab9a, role.id=f1f68385aeb8421b80010aaf19d20d6e, scope.project.id=c9cd84395d7f49969b6df473fd521e6d) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:05.721418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:05.722893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:05.731633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 155/324] 10.222.0.66 () {56 vars in 1376 bytes} [Fri Oct 28 19:47:05 2022] GET /identity/v3/role_assignments?group.id=9dc827da7ca448df96b04026ba26ab9a&scope.project.id=c9cd84395d7f49969b6df473fd521e6d&role.id=f1f68385aeb8421b80010aaf19d20d6e => generated 611 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:05.767919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.768609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.768898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.769175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.783532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:05.789787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:05.790881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:05.791666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 170/325] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:05.797189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:05.811549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:05.882761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:05.949675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a4aa97-b8a5-4814-b8f2-0994ae9f3b83 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:05.992455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:05.993076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:05.993398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:05.993647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:05.994313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:05.994615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:05.995994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.008854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 156/326] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:06.011367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.015201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.032922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de39303-d9ee-479f-bd6c-ac688affb5b1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.072690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.073641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.074092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.074767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.087978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:06.094325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:06.095377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:06.096384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:06.097452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.098495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 171/327] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:06.106117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.146361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bd2818-a54b-4e43-bd60-afb4cb321e8c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.186266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d794664-7845-460a-8688-707ab604960b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.186890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d794664-7845-460a-8688-707ab604960b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.187174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d794664-7845-460a-8688-707ab604960b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.188102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d794664-7845-460a-8688-707ab604960b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.222100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63953151-1d9b-4470-87bb-2f2b62fc1310 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.261049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.262307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.262307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.262570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.263235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:06.264162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.265364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.269165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d794664-7845-460a-8688-707ab604960b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d794664-7845-460a-8688-707ab604960b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.274307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 172/328] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:06.279918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.289479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.352127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.392663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 157/329] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 19:47:06.398573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d794664-7845-460a-8688-707ab604960b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.402573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6e5875-9fd1-4c6d-8a60-5d9a022aa812 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.441119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.441883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.442170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.442450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.523161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.524638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.528356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d794664-7845-460a-8688-707ab604960b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.533003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.567887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.568585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.568865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.569166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] PATH_INFO: `/v3/projects/c9cd84395d7f49969b6df473fd521e6d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.570346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] RBAC: Authorizing `identity:get_project(project_id=c9cd84395d7f49969b6df473fd521e6d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.571786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.573496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 158/330] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/projects/c9cd84395d7f49969b6df473fd521e6d => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:06.578981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.612726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.627113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 173/331] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 19:47:06.632816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.676905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5ec3e1-76f4-433d-bbc3-9d57447ca0a4 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.711499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.712143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.712424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.712696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] PATH_INFO: `/v3/groups/9dc827da7ca448df96b04026ba26ab9a` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.713822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] RBAC: Authorizing `identity:get_group(group_id=9dc827da7ca448df96b04026ba26ab9a)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.715268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.716824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 159/332] 10.222.0.66 () {56 vars in 1180 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/groups/9dc827da7ca448df96b04026ba26ab9a => generated 226 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:06.722606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.729048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb5f1be-d5c8-4c81-87c4-33c147e3cf14 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.736343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.763075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.763711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.764001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.764276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] PATH_INFO: `/v3/projects/9255f3850e094deb824d58d03e377387` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.765426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] RBAC: Authorizing `identity:get_project(project_id=9255f3850e094deb824d58d03e377387)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.766862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.768485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 174/333] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/projects/9255f3850e094deb824d58d03e377387 => generated 344 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:06.774142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:06.827415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3ed782-3405-4d46-8848-4c847d460ad3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.829697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:06.861541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.862140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.862419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.862688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.863399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] RBAC: Adding query filter params (group.id=9dc827da7ca448df96b04026ba26ab9a, role.id=335ad63ce3214c9aa7e4f4f058f97465, scope.project.id=c9cd84395d7f49969b6df473fd521e6d) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:06.863702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.865132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.871381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 160/334] 10.222.0.66 () {56 vars in 1376 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/role_assignments?group.id=9dc827da7ca448df96b04026ba26ab9a&scope.project.id=c9cd84395d7f49969b6df473fd521e6d&role.id=335ad63ce3214c9aa7e4f4f058f97465 => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:06.872160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb60b9-3e5f-4cb9-a151-5e50000f400f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:06.909919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7da338f-357d-431a-959f-8883de07eeec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:06.910571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da338f-357d-431a-959f-8883de07eeec None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:06.910849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da338f-357d-431a-959f-8883de07eeec None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:06.911120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da338f-357d-431a-959f-8883de07eeec None admin] PATH_INFO: `/v3/groups/a2da0988336e48ecaa5182ca4e870080` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:06.912263 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7da338f-357d-431a-959f-8883de07eeec None admin] RBAC: Authorizing `identity:get_group(group_id=a2da0988336e48ecaa5182ca4e870080)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:06.914559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7da338f-357d-431a-959f-8883de07eeec None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:06.916343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 175/335] 10.222.0.66 () {56 vars in 1180 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/groups/a2da0988336e48ecaa5182ca4e870080 => generated 229 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:06.922363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:07.035227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1301a47-055d-49bf-a11b-80a781615d2d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:07.069605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b0d075-58e4-4386-9b01-c293ec9f6a50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:07.070221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b0d075-58e4-4386-9b01-c293ec9f6a50 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:07.070495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b0d075-58e4-4386-9b01-c293ec9f6a50 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:07.070766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b0d075-58e4-4386-9b01-c293ec9f6a50 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:07.071485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b0d075-58e4-4386-9b01-c293ec9f6a50 None admin] RBAC: Adding query filter params (group.id=a2da0988336e48ecaa5182ca4e870080, role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=9255f3850e094deb824d58d03e377387) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:07.071790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b0d075-58e4-4386-9b01-c293ec9f6a50 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:07.073224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b0d075-58e4-4386-9b01-c293ec9f6a50 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:07.079163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 161/336] 10.222.0.66 () {56 vars in 1376 bytes} [Fri Oct 28 19:47:06 2022] GET /identity/v3/role_assignments?group.id=a2da0988336e48ecaa5182ca4e870080&scope.project.id=9255f3850e094deb824d58d03e377387&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 611 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:09.782312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328d2c67-e751-4205-bd8f-4a8c9eda7dc5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:09.783395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328d2c67-e751-4205-bd8f-4a8c9eda7dc5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:09.783877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328d2c67-e751-4205-bd8f-4a8c9eda7dc5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:09.785860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 176/337] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:09.792771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cdd229-1c85-4b2a-b522-9bf774f78385 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:09.793717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cdd229-1c85-4b2a-b522-9bf774f78385 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:09.794294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cdd229-1c85-4b2a-b522-9bf774f78385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:09.822802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f4cdd229-1c85-4b2a-b522-9bf774f78385 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:09.824418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f4cdd229-1c85-4b2a-b522-9bf774f78385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:09.863064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:09.921615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 162/338] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:09.938205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2667899-1ff2-4b37-aa8c-5c8ecdde3b36 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:09.938882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2667899-1ff2-4b37-aa8c-5c8ecdde3b36 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:09.939438 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2667899-1ff2-4b37-aa8c-5c8ecdde3b36 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:09.941119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 177/339] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:09.948254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb933ffe-ac51-448a-89c2-03446f67eb95 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:09.948879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb933ffe-ac51-448a-89c2-03446f67eb95 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:09.949395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb933ffe-ac51-448a-89c2-03446f67eb95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:09.976971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-eb933ffe-ac51-448a-89c2-03446f67eb95 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:09.978221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-eb933ffe-ac51-448a-89c2-03446f67eb95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:10.067535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 163/340] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:10.092186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ac07b-fc3a-4ad0-aa79-07022af153dd None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:10.092796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ac07b-fc3a-4ad0-aa79-07022af153dd None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:10.093266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ac07b-fc3a-4ad0-aa79-07022af153dd None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:10.094834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 178/341] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:10.100276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64cff83-3c90-4bb3-85c8-69cfe0c31df5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:10.100741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64cff83-3c90-4bb3-85c8-69cfe0c31df5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:10.101026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64cff83-3c90-4bb3-85c8-69cfe0c31df5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:10.125468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b64cff83-3c90-4bb3-85c8-69cfe0c31df5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:10.126658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b64cff83-3c90-4bb3-85c8-69cfe0c31df5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:10.249802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 164/342] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:10.360725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830ac07b-fc3a-4ad0-aa79-07022af153dd None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:10.470652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:10.480079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64cff83-3c90-4bb3-85c8-69cfe0c31df5 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:10.508276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:10.601350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64cff83-3c90-4bb3-85c8-69cfe0c31df5 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:10.637434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:10.637434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:10.637434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:10.637434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:10.653360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:10.653360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:10.665351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830ac07b-fc3a-4ad0-aa79-07022af153dd None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:10.673349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:10.674324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:10.675175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:10.675175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:10.675175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:10.675175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 165/343] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:10.675175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:10.725914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:10.725914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:10.725914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:10.725914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:10.745347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:10.749353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:10.773347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:10.774408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:10.776443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:10.776443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:10.776443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:10.777114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 179/344] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 417 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:10.778087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:10.787336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:10.817382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e0d98b-3ad8-4d64-abdb-3c0c3ebed177 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:10.864589 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b6449f-fbe6-4503-833e-8859d4b31e93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:10.865235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b6449f-fbe6-4503-833e-8859d4b31e93 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:10.865635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b6449f-fbe6-4503-833e-8859d4b31e93 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:10.865948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b6449f-fbe6-4503-833e-8859d4b31e93 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:10.876856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b6449f-fbe6-4503-833e-8859d4b31e93 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:10.878328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b6449f-fbe6-4503-833e-8859d4b31e93 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-85b6449f-fbe6-4503-833e-8859d4b31e93 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:10.892628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:10.893562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:10.894407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:10.894407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:10.894407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:10.894407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 166/345] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:10.894919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a340604-754d-4aec-a89c-44b552f53c1d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:10.895206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a340604-754d-4aec-a89c-44b552f53c1d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:10.895481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a340604-754d-4aec-a89c-44b552f53c1d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:10.896998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 167/346] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:10.898388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a340604-754d-4aec-a89c-44b552f53c1d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:10.909729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:10.918603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad5b421-3f68-49b6-bcd9-cd601c0694a4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:10.957526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:10.958259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:10.958542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:10.958823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:10.959460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:10.960130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:10.961875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:10.974503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 180/347] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:10.975832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:11.016374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a340604-754d-4aec-a89c-44b552f53c1d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:11.050255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:11.063595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee8b08b-c860-4646-9afd-ffe83b7c4633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:11.064271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8b08b-c860-4646-9afd-ffe83b7c4633 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.064967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8b08b-c860-4646-9afd-ffe83b7c4633 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.065308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8b08b-c860-4646-9afd-ffe83b7c4633 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.066265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8b08b-c860-4646-9afd-ffe83b7c4633 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:11.066577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8b08b-c860-4646-9afd-ffe83b7c4633 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:11.068777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8b08b-c860-4646-9afd-ffe83b7c4633 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:11.083330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 168/348] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:11.084642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067190b-b79f-4b44-a8a6-67c49b9e1ad0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.085292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067190b-b79f-4b44-a8a6-67c49b9e1ad0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.085856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067190b-b79f-4b44-a8a6-67c49b9e1ad0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.092060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2745d1d7-89ef-4d6c-9708-7a138cb0db14 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:11.105519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-1067190b-b79f-4b44-a8a6-67c49b9e1ad0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:11.106809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1067190b-b79f-4b44-a8a6-67c49b9e1ad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:11.128637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:11.129252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.129580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.129852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.130491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:11.130790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:11.132666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:11.144585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 181/349] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:11.145302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:11.163482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:11.193497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 169/350] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:11.201976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067190b-b79f-4b44-a8a6-67c49b9e1ad0 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:11.276222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d427a8e9-8bc7-4e88-ae41-879f0dd077c7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:11.307688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067190b-b79f-4b44-a8a6-67c49b9e1ad0 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:11.312930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:11.313579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.313867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.314145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.314786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:11.316237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:11.326693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:11.344299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f80662-543a-4e54-8199-d035d8d6a60c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:11.344909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f80662-543a-4e54-8199-d035d8d6a60c None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.345191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f80662-543a-4e54-8199-d035d8d6a60c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.345508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f80662-543a-4e54-8199-d035d8d6a60c None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.346144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f80662-543a-4e54-8199-d035d8d6a60c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:11.347669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f80662-543a-4e54-8199-d035d8d6a60c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:11.450890 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 182/351] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:47:11 2022] POST /identity/v3/users => generated 248 bytes in 306 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:11.451835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:11.473358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 170/352] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:47:11 2022] POST /identity/v3/users => generated 246 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:11.473358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c9aa0-4b09-47ef-b60c-8da4027b89a9 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.473358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c9aa0-4b09-47ef-b60c-8da4027b89a9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.473358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c9aa0-4b09-47ef-b60c-8da4027b89a9 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.481376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 171/353] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:11 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:11.482757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2e2693-23dc-47d6-8930-a81ec29fc5b4 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.483044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2e2693-23dc-47d6-8930-a81ec29fc5b4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.486288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2e2693-23dc-47d6-8930-a81ec29fc5b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.516961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6b2e2693-23dc-47d6-8930-a81ec29fc5b4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:11.518139 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6b2e2693-23dc-47d6-8930-a81ec29fc5b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:11.576576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:11.621835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 172/354] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:11.622906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2e2693-23dc-47d6-8930-a81ec29fc5b4 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:11.627407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c28985-d210-4488-b3ea-5f2f382075f1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:11.672417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:11.673451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.674009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.674486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.675401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:11.677890 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:11.743068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2e2693-23dc-47d6-8930-a81ec29fc5b4 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:11.744711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:11.765815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 183/355] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:47:11 2022] POST /identity/v3/users => generated 248 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:11.791809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:11.792895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:11.793449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:11.793922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:11.807437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:11.811012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:11.827595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:11.829065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:11.830311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:11.830311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:11.830311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:11.831856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 173/356] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:11.838074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:11.905555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:12.036992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fd871e-4fbe-445d-ae1b-eee3fd1fba59 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:12.068466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab06979-0fc4-4807-956c-d837949f13fa None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:12.076306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:12.077070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:12.077518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:12.077897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:12.078921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:12.080183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:12.101919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:12.117284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:12.117937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:12.118404 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:12.118607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:12.119276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:12.119574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:12.120258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.sql.core [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 19:47:12.120258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 28 19:47:12.120258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [parameters: {'id': '99b01bae79964108839915b02fc927af', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 28 19:47:12.120258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81614) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 19:47:12.121099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:47:12.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return ref.to_dict() Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:47:12.130442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application yield resource Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise self.value Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application session.commit() Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 19:47:12.131378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.flush() Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application compat.raise_( Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 19:47:12.132294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:47:12.133190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:47:12.137301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [parameters: {'id': '99b01bae79964108839915b02fc927af', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:12.139271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 28 19:47:12.141269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:12.143321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 174/357] 10.222.0.66 () {60 vars in 1131 bytes} [Fri Oct 28 19:47:11 2022] POST /identity/v3/roles => generated 144 bytes in 233 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:12.143463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 184/358] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:12.143823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:12.145035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:12.173087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:12.282292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1158030-007f-4752-ae7b-77666fb642a1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:12.302951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11ca5a-8dfe-4a0e-990c-c11007cfa562 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:12.356539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e1b745-7dfe-4ca2-8beb-a76b934e2fa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:12.357745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1b745-7dfe-4ca2-8beb-a76b934e2fa9 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:12.359006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1b745-7dfe-4ca2-8beb-a76b934e2fa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:12.360044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1b745-7dfe-4ca2-8beb-a76b934e2fa9 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:12.361060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e1b745-7dfe-4ca2-8beb-a76b934e2fa9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:12.363618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e1b745-7dfe-4ca2-8beb-a76b934e2fa9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:12.367361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:12.368292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:12.368789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:12.369271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:12.390831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:12.396078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:12.400993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:12.401882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 175/359] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:12.403717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:12.451912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 185/360] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:47:12 2022] POST /identity/v3/users => generated 249 bytes in 307 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:12.515158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:12.551902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f61c879-af0c-4514-8cc7-1565864992d8 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:12.602215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c059459e-4acd-424d-bb1e-45901fadb120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:12.602854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c059459e-4acd-424d-bb1e-45901fadb120 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:12.603140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c059459e-4acd-424d-bb1e-45901fadb120 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:12.603415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c059459e-4acd-424d-bb1e-45901fadb120 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:12.604102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c059459e-4acd-424d-bb1e-45901fadb120 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:12.604406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c059459e-4acd-424d-bb1e-45901fadb120 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:12.605865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c059459e-4acd-424d-bb1e-45901fadb120 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:12.616584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 176/361] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:13.910858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce888bd-e176-4108-b071-6a7404890d12 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:13.911410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce888bd-e176-4108-b071-6a7404890d12 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:13.911410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce888bd-e176-4108-b071-6a7404890d12 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:13.913175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 186/362] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:13.918382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f8b5e8-25e8-4211-90e6-50a4ee7ada98 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:13.918796 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f8b5e8-25e8-4211-90e6-50a4ee7ada98 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:13.918979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f8b5e8-25e8-4211-90e6-50a4ee7ada98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:13.943030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d2f8b5e8-25e8-4211-90e6-50a4ee7ada98 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:13.944695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d2f8b5e8-25e8-4211-90e6-50a4ee7ada98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:14.034778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 177/363] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:14.134580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef705a1-4b66-4220-be70-95173d63554b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:14.135121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef705a1-4b66-4220-be70-95173d63554b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:14.135542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef705a1-4b66-4220-be70-95173d63554b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:14.137024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 187/364] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:14.142291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b9bfe-b47b-486c-9b27-f60016624e3a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:14.142585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b9bfe-b47b-486c-9b27-f60016624e3a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:14.142854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b9bfe-b47b-486c-9b27-f60016624e3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:14.165919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c44b9bfe-b47b-486c-9b27-f60016624e3a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:14.167125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c44b9bfe-b47b-486c-9b27-f60016624e3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:14.255296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:14.259831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 178/365] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:14.363651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef705a1-4b66-4220-be70-95173d63554b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:14.417308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:14.498038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef705a1-4b66-4220-be70-95173d63554b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:14.534273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44b9bfe-b47b-486c-9b27-f60016624e3a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:14.542530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:14.543629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:14.543924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:14.544205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:14.567603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:14.576861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:14.577817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:14.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:14.582321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 188/366] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:14.588174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:14.718220 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44b9bfe-b47b-486c-9b27-f60016624e3a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:14.781234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c04cf2-2dd5-4e94-9e83-d0c71d6e14d5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:14.782954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:14.789734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:14.790660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:14.791185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:14.792110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:14.804588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:14.811245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:14.812266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:14.813059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 179/367] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:14.818616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:14.861772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffcec6e-ddae-4f96-b0a3-da587a6cfa4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:14.867499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffcec6e-ddae-4f96-b0a3-da587a6cfa4e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:14.868207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffcec6e-ddae-4f96-b0a3-da587a6cfa4e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:14.868826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffcec6e-ddae-4f96-b0a3-da587a6cfa4e None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:14.873922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffcec6e-ddae-4f96-b0a3-da587a6cfa4e None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:14.873922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffcec6e-ddae-4f96-b0a3-da587a6cfa4e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:14.873922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffcec6e-ddae-4f96-b0a3-da587a6cfa4e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:14.885740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 189/368] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:14.887208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54b786-2eb3-462a-9c5c-f89b0d4f0d49 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:14.887656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54b786-2eb3-462a-9c5c-f89b0d4f0d49 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:14.888079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54b786-2eb3-462a-9c5c-f89b0d4f0d49 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:14.889789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 190/369] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:14.894571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e759c-2b6f-4f24-9ebe-0063d775a14e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:14.895124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e759c-2b6f-4f24-9ebe-0063d775a14e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:14.895575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e759c-2b6f-4f24-9ebe-0063d775a14e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:14.902197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:14.945407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050faf2-d0d6-4e57-a55a-8a7c1d7618f3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:14.995840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3d0e759c-2b6f-4f24-9ebe-0063d775a14e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:14.995840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3d0e759c-2b6f-4f24-9ebe-0063d775a14e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:15.022826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.023960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.024713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.025162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.026039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] RBAC: Adding query filter params (name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:15.026542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.028141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:15.043550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 180/370] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:15.045271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.098992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 191/371] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:47:15.100552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e759c-2b6f-4f24-9ebe-0063d775a14e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.198760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d464204f-ddbe-454e-97a2-7aabd954064e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.238339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:15.253562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674c5c01-5cd4-4495-a442-1f9af172a5f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.253562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674c5c01-5cd4-4495-a442-1f9af172a5f8 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.253562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674c5c01-5cd4-4495-a442-1f9af172a5f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.253562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674c5c01-5cd4-4495-a442-1f9af172a5f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.263113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674c5c01-5cd4-4495-a442-1f9af172a5f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.264783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674c5c01-5cd4-4495-a442-1f9af172a5f8 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-674c5c01-5cd4-4495-a442-1f9af172a5f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:15.290070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:15.291029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:15.291866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:15.291866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:15.291866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:15.294090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 181/372] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:15.295484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:15.299722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc97966-9989-4b6b-8cec-c5911056e4fe None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.300020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc97966-9989-4b6b-8cec-c5911056e4fe None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.300300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc97966-9989-4b6b-8cec-c5911056e4fe None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.301247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e759c-2b6f-4f24-9ebe-0063d775a14e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.306021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 182/373] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:15.306021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc97966-9989-4b6b-8cec-c5911056e4fe None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.352056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a752747-9dd4-4131-bdd7-928789861af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.352714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a752747-9dd4-4131-bdd7-928789861af5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.353000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a752747-9dd4-4131-bdd7-928789861af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.353276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a752747-9dd4-4131-bdd7-928789861af5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.368516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a752747-9dd4-4131-bdd7-928789861af5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.370326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a752747-9dd4-4131-bdd7-928789861af5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8a752747-9dd4-4131-bdd7-928789861af5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:15.389739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:15.390700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:15.391539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:15.391539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:15.391539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:15.393276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 192/374] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:15.395023 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1771fe90-7257-44d3-8e73-3dac249924e4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.395501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1771fe90-7257-44d3-8e73-3dac249924e4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.395964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1771fe90-7257-44d3-8e73-3dac249924e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.419718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1771fe90-7257-44d3-8e73-3dac249924e4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:15.420950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1771fe90-7257-44d3-8e73-3dac249924e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:15.431218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc97966-9989-4b6b-8cec-c5911056e4fe None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.470631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.471238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.471521 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.471795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.472433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:15.472738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.474142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:15.492314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 183/375] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:15.492314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.505038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 193/376] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:15.510234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1771fe90-7257-44d3-8e73-3dac249924e4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.525646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:15.596618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a90896-a187-44c3-8d73-f80fc03fcfe3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.602410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:15.628585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1771fe90-7257-44d3-8e73-3dac249924e4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.632117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.632733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.633024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.633303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.633976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:15.634282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.635671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:15.649210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 184/377] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:15.650314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.668801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920a580-d524-4053-86d8-e5f08a071210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.669793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920a580-d524-4053-86d8-e5f08a071210 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.670281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920a580-d524-4053-86d8-e5f08a071210 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.670659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920a580-d524-4053-86d8-e5f08a071210 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.682511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e920a580-d524-4053-86d8-e5f08a071210 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e920a580-d524-4053-86d8-e5f08a071210 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:15.688066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:15.689032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 194/378] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:15.690309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac5f8d2-d233-4ced-addf-dcb2a695006d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.691056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac5f8d2-d233-4ced-addf-dcb2a695006d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.691056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac5f8d2-d233-4ced-addf-dcb2a695006d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.691884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 195/379] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:15 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:15.696828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac5f8d2-d233-4ced-addf-dcb2a695006d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.752411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:15.760010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1499ad-93be-4e38-968a-37159c2a8bf9 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.795534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.796165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.796447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.796728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.805532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac5f8d2-d233-4ced-addf-dcb2a695006d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.805796 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.807261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:15.821196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:15.827028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 185/380] 10.222.0.66 () {56 vars in 1093 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:15.828007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.847052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99100f3e-ba37-42cb-b5f5-782bfec437b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.847686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99100f3e-ba37-42cb-b5f5-782bfec437b0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.847973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99100f3e-ba37-42cb-b5f5-782bfec437b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.848242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99100f3e-ba37-42cb-b5f5-782bfec437b0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.851848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:15.859049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99100f3e-ba37-42cb-b5f5-782bfec437b0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.860592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99100f3e-ba37-42cb-b5f5-782bfec437b0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-99100f3e-ba37-42cb-b5f5-782bfec437b0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:15.874348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:15.875495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 196/381] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/users/glance => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:15.876630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94d0d00-05e4-46e2-a786-7597d35e4910 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.877245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94d0d00-05e4-46e2-a786-7597d35e4910 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.877245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94d0d00-05e4-46e2-a786-7597d35e4910 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.895403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f94d0d00-05e4-46e2-a786-7597d35e4910 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:15.896628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f94d0d00-05e4-46e2-a786-7597d35e4910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:15.937770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610bd420-90f6-4cf9-bf3c-51bd4eb04298 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:15.965748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 197/382] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:15.966568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94d0d00-05e4-46e2-a786-7597d35e4910 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:15.983496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:15.984171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:15.984390 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:15.984662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:15.985348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] RBAC: Adding query filter params (name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:15.985748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:15.987146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.000314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 186/383] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:16.000314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.022577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.089750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94d0d00-05e4-46e2-a786-7597d35e4910 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.111836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e22bac-4dbe-47fc-8933-4cf3560a4ebd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.117235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.127018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.127668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.127954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.128228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.139279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:16.144236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:16.145119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:16.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:16.146575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 198/384] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:16.148301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.152361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.153268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.153710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.154191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.155138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:16.155586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.157433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.199477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 187/385] 10.222.0.66 () {56 vars in 1133 bytes} [Fri Oct 28 19:47:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:16.200598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.251495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bbcc81-7c1a-4a67-ba4e-6293b8a2a7a9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.278337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.285970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.286590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.294894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.294894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.294894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:16.294894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.294894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.302163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031bed3-2c31-4623-94e7-6aa06526b8cf None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.327259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 199/386] 10.222.0.66 () {56 vars in 1137 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:16.328093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.338232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.338899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.339183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.339458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.349344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.350855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:16.365535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:16.366522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:16.372300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:16.372300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.372300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:16.372300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 188/387] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:16.372300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.372771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.427713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9710c6ea-0604-4a24-92cc-d2a0429cb351 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.461944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.462557 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.462842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.463128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.463794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:16.464104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.465514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.468830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9c9a76-5a10-4bbc-bb00-b3caa928f751 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.473444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 200/388] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:16.474145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.496611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.505739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.507181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.507181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.507493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.519147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.520614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:16.535575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:16.536500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:16.537407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:16.537407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.537407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:16.537407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 189/389] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:16.537407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.577179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab7d9d0-849f-48cc-b3ad-60f5eb982794 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.583851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.611084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.611757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.612175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.612804 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.624817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.626379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.640292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8717ee-9cde-447d-a6f2-a45608a2217f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:16.641032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:16.642007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:16.643295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:16.643295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.643295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:16.643295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 201/390] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:16.643295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.676453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.677077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.677387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.677652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.688342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.689822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:16.703671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:16.704603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:16.705494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:16.705494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.705494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:16.709125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 190/391] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:16.710808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.740040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.743959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aada5e9-68d3-40f0-bae6-eed350e8e496 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.777711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.778333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.778625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.778902 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.779550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:16.779854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.781229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.793347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 202/392] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:16.794120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.812707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.813985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d81f5fc-d373-4c33-b2ec-c6244bc78301 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.847294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.847916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.848227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.848502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.859044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.860991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:16.875509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:16.876428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:16.877271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:16.877271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:16.877271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:16.877271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 191/393] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:16.877881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.899779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c6521d-22cf-4fe9-8ed2-642e1c91a71c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:16.935828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:16.936436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:16.936718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:16.936993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:16.937652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:16.937960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:16.939316 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:16.951030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 203/394] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:16.951730 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:16.963060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:16.981150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c43c1-c26a-4ac9-b51d-d0693f989e05 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.018231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.018935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.019228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.019506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.020151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:17.020456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.021883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.032078 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.034281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 192/395] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:17.035097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.054707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba140a09-b80b-46b4-928f-91b6821b1eca None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.088789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.089491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.089779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.090309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.090676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:17.090980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.092364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.104436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 204/396] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:17.105231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.133361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61feb5c7-d54d-4744-a949-67533b9288f0 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.166868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.167502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.167782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.168056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.174918 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.176372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:17.188845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:17.189772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:17.190668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 193/397] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/users/cinder => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:17.191440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.193815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.208828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4158a42d-ccf5-48fd-96c6-f35c7d9346ac None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.246026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.246646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.246931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.247210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.247848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:17.248152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.249571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.255823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.261492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 205/398] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:17.262287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.296603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e89c2ea-eaa9-4c37-8051-7bf59baa1167 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.329790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.330353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.330637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.330914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.338190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.339661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:17.351480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:17.352500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:17.353610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 194/399] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:17.354211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.362441 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c7a9e-a77e-47d4-b801-1ab268334cdd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.397509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.398175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.398459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.398733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.399363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.400792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.415638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.457511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4996dc7b-9c57-472a-8dcb-cec84a788209 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.474043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 206/400] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:47:17 2022] POST /identity/v3/users => generated 247 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:17.475143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.497312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.498042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.498329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.498603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.506399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.508651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:17.521670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:17.522971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:17.524130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:17.525518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 195/401] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:17.526570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.547965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.599367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c04c1e-e583-4dc8-8dad-7a5814893ac7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.636706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.637535 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.637926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.638310 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.639181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:17.639597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.639934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff103ff-11c9-44a0-baee-1998d2b4d6fb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.641256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.661178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 207/402] 10.222.0.66 () {56 vars in 1137 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:17.662399 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.671879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.686252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.686886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.687218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.687497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.696638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.698249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:17.710992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:17.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:17.713113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 196/403] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/users/neutron => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:17.713755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.777868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e4b152-2379-4925-932b-e56d8840c48d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.791891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.820851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.821761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.822055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.822335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.823025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:17.823327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.824750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.832519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfd63bc-f05f-4dab-981a-d42e5589c22a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.845032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 208/404] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:17.846369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.870774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.871457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.871740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.872013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:17.872708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:17.873009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:17.874637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:17.908610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 197/405] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:17.909386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:17.955951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d02b17-6912-4aa3-bae6-afcfa8d3d922 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:17.957077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:17.993500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:17.994305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:17.994688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:17.995069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.005573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.007714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:18.023646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:18.024756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:18.025799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:18.025799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:18.025799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:18.031611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 209/406] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:18.032133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605dc51-c516-479b-89ff-8a1d4aca2c19 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.042333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.060751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:18.080365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.081164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.081572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.081955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.082837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:18.083336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.085480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.104182 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 198/407] 10.222.0.66 () {56 vars in 1139 bytes} [Fri Oct 28 19:47:17 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:18.104924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.143495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8039a89d-4ef0-4fa6-a8d4-0be5fd73bb6a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.182243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.183285 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.183574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.183854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/d54e8c38f0f84e4287b0445947bde916/roles/57b3843b68f846d5905474ee762f7059` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.199597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=d54e8c38f0f84e4287b0445947bde916, role_id=57b3843b68f846d5905474ee762f7059)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.201066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.203285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:18.206859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ec0664-e84b-4f39-a31a-69fcb316c23d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.241990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.242377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.242653 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.242942 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/a7fae443ad92439aadd2f94ad3bca594/roles/57b3843b68f846d5905474ee762f7059` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.258086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=a7fae443ad92439aadd2f94ad3bca594, role_id=57b3843b68f846d5905474ee762f7059)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.259533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.268056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 210/408] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:18 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/d54e8c38f0f84e4287b0445947bde916/roles/57b3843b68f846d5905474ee762f7059 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:18.268891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.333972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 199/409] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:18 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/a7fae443ad92439aadd2f94ad3bca594/roles/57b3843b68f846d5905474ee762f7059 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:18.336175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.398199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:18.456948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa60a07-dada-4b70-8a4e-74a0a68cb8dd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.509296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.509972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.510263 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.510544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.511216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:18.511521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.512932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.530810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 211/410] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:18.537599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.543805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd7b62-8973-4366-82ce-daeafe9999e9 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.589991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.590871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.591327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.591745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.605510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.607030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:18.625876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:18.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:18.628043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:18.628043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:18.628043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:18.631777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 200/411] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:18.633187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.636787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:18.676937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b6d54-fa68-47ba-9d67-e56707731f7d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.715392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.716093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.716385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.716666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.718931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:18.724571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.727214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:18.739907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:18.741587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:18.742431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:18.746665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 212/412] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:18.747985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.760883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec61ea-e361-4df8-b9bd-0832ebdccfbd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.815561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.816196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.816484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.816763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.817904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:18.818044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.819527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.853402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 201/413] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:18.859961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:18.902198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ccfc99-7295-4858-987d-7ed274eaf56a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:18.968799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:18.969565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:18.969858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:18.970139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:18.970942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:18.971258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:18.977345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:18.991779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:18.992117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 213/414] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:18.993357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:19.056217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:19.086617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0122e512-45ba-4df2-a0af-f120e7539a1f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:19.125044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e65b7-ee7c-4c49-a4fd-9a7ea9a720ee None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:19.162211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:19.163059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:19.163346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:19.163678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/183a4f23782143b0b234d7800b409452/roles/57b3843b68f846d5905474ee762f7059` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:19.179108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:19.179750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:19.180026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:19.180296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:19.180700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=183a4f23782143b0b234d7800b409452, role_id=57b3843b68f846d5905474ee762f7059)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:19.182214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:19.195743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:19.197345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:19.209940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:19.210825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:19.211664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:19.216281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 202/415] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:19.217556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:19.256600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 214/416] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:18 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/183a4f23782143b0b234d7800b409452/roles/57b3843b68f846d5905474ee762f7059 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:19.367618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2597434-913b-47fe-9699-ed8173facf8f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:19.395194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:19.416887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65305d43-1e34-4104-871c-940c20c4446f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:19.417553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65305d43-1e34-4104-871c-940c20c4446f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:19.417837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65305d43-1e34-4104-871c-940c20c4446f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:19.418116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65305d43-1e34-4104-871c-940c20c4446f None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:19.418833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65305d43-1e34-4104-871c-940c20c4446f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:19.419142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65305d43-1e34-4104-871c-940c20c4446f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:19.420608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65305d43-1e34-4104-871c-940c20c4446f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:19.439426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 203/417] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:19.446464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:19.592733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b84bdd-6d4a-4433-994a-50a9f853fff9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:19.595511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:19.628892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d74450-5362-4d2f-8361-f33cb5a549e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:19.630035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d74450-5362-4d2f-8361-f33cb5a549e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:19.630565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d74450-5362-4d2f-8361-f33cb5a549e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:19.631022 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d74450-5362-4d2f-8361-f33cb5a549e0 None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/91639dc527ae47e892164544572cbec4/roles/57b3843b68f846d5905474ee762f7059` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:19.649294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d74450-5362-4d2f-8361-f33cb5a549e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=91639dc527ae47e892164544572cbec4, role_id=57b3843b68f846d5905474ee762f7059)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:19.651218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d74450-5362-4d2f-8361-f33cb5a549e0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:19.732197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 215/418] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:19 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/91639dc527ae47e892164544572cbec4/roles/57b3843b68f846d5905474ee762f7059 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:19.773751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed3bbc3-8c67-4da6-a66d-061b5db9870a None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:19.774039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed3bbc3-8c67-4da6-a66d-061b5db9870a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:19.774319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed3bbc3-8c67-4da6-a66d-061b5db9870a None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:19.775724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 204/419] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:19.781395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dba8db-7f97-4bfe-9f1e-b39beafd178e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:19.782032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dba8db-7f97-4bfe-9f1e-b39beafd178e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:19.782492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dba8db-7f97-4bfe-9f1e-b39beafd178e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:19.805590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-34dba8db-7f97-4bfe-9f1e-b39beafd178e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:19.807182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-34dba8db-7f97-4bfe-9f1e-b39beafd178e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:19.909860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 216/420] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:47:20.260612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed3bbc3-8c67-4da6-a66d-061b5db9870a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:20.379941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed3bbc3-8c67-4da6-a66d-061b5db9870a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:20.421642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:20.422638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:20.423121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:20.423589 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:20.441562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:20.446856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:20.447746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:20.448635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:20.450968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 205/421] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:20.462466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dba8db-7f97-4bfe-9f1e-b39beafd178e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:20.551530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:20.577924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dba8db-7f97-4bfe-9f1e-b39beafd178e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:20.626100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:20.626731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:20.627018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:20.627300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:20.627981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:20.628287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:20.629771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:20.639651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 217/422] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:20.645004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:20.721697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:20.782167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0071ba46-074d-44c9-bede-6221b0dabf39 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:20.826145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:20.827138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:20.827623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:20.828119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:20.839496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:20.841252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:20.861654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:20.862665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:20.863519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:20.863519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:20.863519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:20.865618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 206/423] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:20.874468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.007849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d829ed-008c-40e9-a7a2-de36a1b47d27 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:21.048345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec14d217-53d4-420a-ab14-4af7c82f4d22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:21.048968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14d217-53d4-420a-ab14-4af7c82f4d22 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.049253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14d217-53d4-420a-ab14-4af7c82f4d22 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.049561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14d217-53d4-420a-ab14-4af7c82f4d22 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.050212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec14d217-53d4-420a-ab14-4af7c82f4d22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:21.050516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec14d217-53d4-420a-ab14-4af7c82f4d22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:21.051907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec14d217-53d4-420a-ab14-4af7c82f4d22 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:21.073902 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 218/424] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:21.081258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.192382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126561d7-1f75-4f73-a9aa-09531630995d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.192724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126561d7-1f75-4f73-a9aa-09531630995d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.192947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126561d7-1f75-4f73-a9aa-09531630995d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.195485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 219/425] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:47:21.196810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141f647f-77cb-49ea-8c25-0c86482ef367 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.197229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141f647f-77cb-49ea-8c25-0c86482ef367 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.197546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141f647f-77cb-49ea-8c25-0c86482ef367 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.199048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 220/426] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:21.201065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3daadb-58c2-45f7-b022-fa1db2a1d714 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.201379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3daadb-58c2-45f7-b022-fa1db2a1d714 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.201661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3daadb-58c2-45f7-b022-fa1db2a1d714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.215831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:21.222872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ea3daadb-58c2-45f7-b022-fa1db2a1d714 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:21.224275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ea3daadb-58c2-45f7-b022-fa1db2a1d714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:21.229386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:21.236399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fd8b5c-8c04-44a6-b286-1860c142af85 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:21.273379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3a71e3-f691-454b-946d-9eee957a9a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:21.274016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3a71e3-f691-454b-946d-9eee957a9a0a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.274239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3a71e3-f691-454b-946d-9eee957a9a0a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.274505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3a71e3-f691-454b-946d-9eee957a9a0a None admin] PATH_INFO: `/v3/users/swift` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.282475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3a71e3-f691-454b-946d-9eee957a9a0a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:21.283939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3a71e3-f691-454b-946d-9eee957a9a0a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-1c3a71e3-f691-454b-946d-9eee957a9a0a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:21.296124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:21.297254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:21.298396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:21.301940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 207/427] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:21 2022] GET /identity/v3/users/swift => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:21.303745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b85bd1-2eb1-4abb-b787-fda47d8bcb1d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.304049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b85bd1-2eb1-4abb-b787-fda47d8bcb1d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.304325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b85bd1-2eb1-4abb-b787-fda47d8bcb1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.317913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 221/428] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:21.319321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3daadb-58c2-45f7-b022-fa1db2a1d714 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.323438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-79b85bd1-2eb1-4abb-b787-fda47d8bcb1d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:21.324111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-79b85bd1-2eb1-4abb-b787-fda47d8bcb1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:21.393726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 208/429] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:21.430402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3daadb-58c2-45f7-b022-fa1db2a1d714 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:21.468048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:21.469492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.470034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.470531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.471458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:21.471982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:21.473774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:21.477472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:21.494275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 222/430] 10.222.0.66 () {56 vars in 1135 bytes} [Fri Oct 28 19:47:21 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:21.501366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b85bd1-2eb1-4abb-b787-fda47d8bcb1d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.602743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:21.625126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b85bd1-2eb1-4abb-b787-fda47d8bcb1d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:21.668831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:21.669421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.669706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.669982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.680597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:21.683178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:21.688640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:21.704229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:21.705558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:21.706500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:21.706500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:21.706500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:21.706500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 209/431] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:21.712905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.825164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b047e-bb51-489b-ac10-a1d551f2b090 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:21.828489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b7b6c0-9571-4968-a6a2-b5ff6e26dbee None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:21.860244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:21.861092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.861548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.861986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.862768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:21.863211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:21.864909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:21.868239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:21.869972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:21.870851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:21.871360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:21.871814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:21.879209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 210/432] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:21.879878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.885840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:21.891099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:21.891980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:21.892777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:21.895474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 223/433] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:21.896501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:21.965951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:21.989285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e8327a-9090-4e37-96fd-96f8a961c214 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.024396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47150d26-b162-4106-9734-0d52e6a522f1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.025188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.028282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.028282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.028282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.037370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:22.042694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:22.043778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:22.044776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:22.047090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 211/434] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:22.048128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.083530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.084133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.084947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.085241 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.093940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.095411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:22.108614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:22.109530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:22.110376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 224/435] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:22.110838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.122449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:22.167391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb209316-8f76-4094-b8dc-f3982c0398cd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.203594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.204213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.204495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.204770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.205457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] RBAC: Adding query filter params (name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:22.205761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.207136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.214990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 212/436] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:22.215802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.230051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0255f6a9-1d89-43b8-81d9-7e009468439e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.246679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:22.268323 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9b6650-376f-4e00-bf9e-8082fbc0d4de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.269291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9b6650-376f-4e00-bf9e-8082fbc0d4de None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.269670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9b6650-376f-4e00-bf9e-8082fbc0d4de None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.269878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9b6650-376f-4e00-bf9e-8082fbc0d4de None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.270624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9b6650-376f-4e00-bf9e-8082fbc0d4de None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:22.270841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9b6650-376f-4e00-bf9e-8082fbc0d4de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.272240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9b6650-376f-4e00-bf9e-8082fbc0d4de None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.285463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 225/437] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:22.285463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9514dba-4ead-434c-92e8-cd3460230b8d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.285463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9514dba-4ead-434c-92e8-cd3460230b8d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.285463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9514dba-4ead-434c-92e8-cd3460230b8d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.285463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 226/438] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:22.285463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9514dba-4ead-434c-92e8-cd3460230b8d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.333637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25646ce3-24f6-4e73-bbea-58b4d51fce27 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.352207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:22.370280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.370897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.371193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.371452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.372164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:22.372555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.374039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.387311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 213/439] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:22.387967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.404205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9514dba-4ead-434c-92e8-cd3460230b8d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.439116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8183cf7e-4e73-44cb-a046-dced08b53f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.439749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8183cf7e-4e73-44cb-a046-dced08b53f80 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.440030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8183cf7e-4e73-44cb-a046-dced08b53f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.440440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8183cf7e-4e73-44cb-a046-dced08b53f80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.452015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8183cf7e-4e73-44cb-a046-dced08b53f80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.453477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8183cf7e-4e73-44cb-a046-dced08b53f80 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8183cf7e-4e73-44cb-a046-dced08b53f80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:22.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:22.469207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:22.470156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:22.470156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:22.470156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:22.470156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 227/440] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:22.470156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ec61a-14ba-4c00-9c7f-b6730e811041 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.470415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ec61a-14ba-4c00-9c7f-b6730e811041 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.470653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ec61a-14ba-4c00-9c7f-b6730e811041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.488020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5f8ec61a-14ba-4c00-9c7f-b6730e811041 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:22.489112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5f8ec61a-14ba-4c00-9c7f-b6730e811041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:22.492347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa6e691-45a8-4b0d-ac5c-0fd0a4488d5f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.513018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:22.526448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fac573-74c5-4569-b338-ac1bc9f2b45a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.527079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fac573-74c5-4569-b338-ac1bc9f2b45a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.527360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fac573-74c5-4569-b338-ac1bc9f2b45a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.527673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fac573-74c5-4569-b338-ac1bc9f2b45a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.538760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fac573-74c5-4569-b338-ac1bc9f2b45a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.540221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fac573-74c5-4569-b338-ac1bc9f2b45a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-f5fac573-74c5-4569-b338-ac1bc9f2b45a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:22.555670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:22.556663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:22.557544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:22.557544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:22.557544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:22.557704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 228/441] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:22.557772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 214/442] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:22.557835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8ec61a-14ba-4c00-9c7f-b6730e811041 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.558196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0596a0f5-1d1e-4d81-a0b1-e32390334cba None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.558479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0596a0f5-1d1e-4d81-a0b1-e32390334cba None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.558753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0596a0f5-1d1e-4d81-a0b1-e32390334cba None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.561431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 215/443] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:22.561431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0596a0f5-1d1e-4d81-a0b1-e32390334cba None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.599645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:22.666242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8ec61a-14ba-4c00-9c7f-b6730e811041 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.681826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0596a0f5-1d1e-4d81-a0b1-e32390334cba None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.700822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.701520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.701805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.702081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/0df958d5d645473c9c0403bf9b4ee846/roles/57b3843b68f846d5905474ee762f7059` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.716901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=0df958d5d645473c9c0403bf9b4ee846, role_id=57b3843b68f846d5905474ee762f7059)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.718369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.721786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f99e14d-f883-4c2c-bfc8-2dcfb5965e4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.722399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f99e14d-f883-4c2c-bfc8-2dcfb5965e4a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.722679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f99e14d-f883-4c2c-bfc8-2dcfb5965e4a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.722952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f99e14d-f883-4c2c-bfc8-2dcfb5965e4a None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.723587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f99e14d-f883-4c2c-bfc8-2dcfb5965e4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:22.723892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f99e14d-f883-4c2c-bfc8-2dcfb5965e4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.725945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f99e14d-f883-4c2c-bfc8-2dcfb5965e4a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.736904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 216/444] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:22.742080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6365e88-2d59-4675-a136-c34ac994dd14 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.742080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6365e88-2d59-4675-a136-c34ac994dd14 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.742080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6365e88-2d59-4675-a136-c34ac994dd14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.755988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e6365e88-2d59-4675-a136-c34ac994dd14 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:22.757499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e6365e88-2d59-4675-a136-c34ac994dd14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:22.776913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 229/445] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:22 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/0df958d5d645473c9c0403bf9b4ee846/roles/57b3843b68f846d5905474ee762f7059 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:22.778046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.795463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:22.855080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 217/446] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:22.856299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6365e88-2d59-4675-a136-c34ac994dd14 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:22.918402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5889230f-74f7-4f22-8c53-1e51fcf58512 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.925740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:22.967984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:22.968872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:22.969299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6365e88-2d59-4675-a136-c34ac994dd14 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:22.969428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:22.969844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:22.970618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:22.971047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:22.973086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:22.987521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 230/447] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:22.988673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.021092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.021693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.022080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.022358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.035006 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.036463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:23.049490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:23.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 218/448] 10.222.0.66 () {56 vars in 1093 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:23.051805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.079919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.116732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ea3673-1552-44de-9cae-0bebfc1250a4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.152203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.152839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.153121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.153428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.158152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.164522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:23.170719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:23.174305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 231/449] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:23.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.183820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c0ac48-bbe7-485b-af3d-8064bea10b65 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.222159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.222827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.223137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.223412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] PATH_INFO: `/v3/users/glance` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.232771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.234281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:23.246318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:23.247493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 219/450] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/users/glance => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:23.248404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.286746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a53ac0-15c9-4ddb-9662-267630f7c605 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.323543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.333014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.333657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.333938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.334964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.335672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:23.335993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.344122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.359708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 232/451] 10.222.0.66 () {56 vars in 1133 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:23.359708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.373166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0437ba-89f9-4888-85cc-af363a64e22d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.411685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.412326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.412657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.412960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.420954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.425584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:23.431098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:23.434510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 220/452] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:23.435765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.475556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c8ab95-ecb7-4e50-a13a-f40d2053dd2a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.512047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.512666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.513051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.513348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.514030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:23.514342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.515726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.529295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 233/453] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:23.529295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.551270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05db5f1d-fc59-4d4a-9c12-ca3ac70b24dd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.582334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.590389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.591004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.591288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.591564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.595946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:23.595946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.595946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.612152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 221/454] 10.222.0.66 () {56 vars in 1137 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:23.613102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.642529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78b0f19-16e0-4486-837e-2c628805abf2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.674659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.680656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.681272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.681588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.681866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.726219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251de737-754a-4fbd-8d6f-85ea32f9ccda None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.760429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.762014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.766549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.767169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.767454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.767735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.768393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] RBAC: Adding query filter params (name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:23.768699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.770599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.786263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 222/455] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:23.786263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.813482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.875084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 234/456] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:47:23.875968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.889479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b589d2-3d9f-4816-bb99-e6f37179c90f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:23.908944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:23.922802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:23.923434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:23.923716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:23.923990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:23.934670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:23.936494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:23.953446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:23.954367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:23.955212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:23.955212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:23.955212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:23.955212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 223/457] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:23.955849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:23.991848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd0c5ee-dd05-45af-af26-75a4cabed96c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.040125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.040748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.041036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.041315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.060763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:24.073161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ebcd07-8b18-44a1-b07c-f0ae13f71221 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.111003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.111645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.111929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.112241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.116978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.118538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.124961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.126455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:24.146088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:24.147222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:24.148078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:24.148078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:24.148078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:24.148078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 224/458] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:24.148836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.188051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:24.246290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 235/459] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:47:24.246290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.272892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba2e7e9-89dd-41d6-b291-dbbd34b73d84 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.298376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:24.319569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.320207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.320491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.325256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.332508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.334056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:24.352231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:24.353235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:24.354559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:24.354559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:24.354559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:24.354559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 225/460] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:24.354559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.372908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c08512-2950-413d-a504-7f639a9ef0bd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.422393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.423018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.423302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.423578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.424220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:24.424528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.425941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.441050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 236/461] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:24.442395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.473730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:24.500162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b464f81-89ff-41af-bb71-3f2d265a2731 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.549869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.550483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.550765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.551317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.551697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:24.552087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.554126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.572033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 226/462] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:24.572033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.572309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9187d22-8966-47f6-9392-7bbe6141628e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.583759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:24.614600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.615489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.616355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.617202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.635461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.637439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:24.655951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:24.657012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:24.658429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:24.658429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:24.658429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:24.659849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 237/463] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:24.665395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.709240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8081647-cb24-4686-8b21-9070c325f195 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.749190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.749843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.750124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.750409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.751046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:24.751351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.752922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.755335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:24.771836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 227/464] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:24.773089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.807374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e718a-7df8-49c6-916a-380a506dd3a2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.846818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.847456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.847741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.848067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.857208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.858754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:24.871390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:24.872356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 238/465] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/users/cinder => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:24.877466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:24.904621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:24.909724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28205f92-e51d-4473-afc8-a8fb78894a8a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:24.957644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:24.958287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:24.958571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:24.958886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:24.969260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:24.971663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:24.989697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:24.990990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 19:47:24.992043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:24.998422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 228/466] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/users/neutron => generated 85 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:24.998422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.014256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b17f85-1987-4a09-9525-751dd1f8b03e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.067490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.068123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.068412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.068694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.069712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:25.070088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.071520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.088771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 239/467] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:25.090401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.099210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.155661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c48cf46-3eb0-48ff-a5d1-190d674f207b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.195705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.196333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.196614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.196897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.205239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.207769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.208567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:25.221488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:25.230122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 229/468] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:25.231866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.238043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5313b977-7bff-4fad-bea8-fd8f7c95b7de None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.286407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.287019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.287300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.287581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.288295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:25.288598 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.290149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.312255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 240/469] 10.222.0.66 () {56 vars in 1137 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:25.312255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.336671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa70a8-eac8-4930-b97e-ad972b540cdd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.352322 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.369729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49595cef-3293-4bdb-b2e4-b188248f06c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.370339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49595cef-3293-4bdb-b2e4-b188248f06c2 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.370618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49595cef-3293-4bdb-b2e4-b188248f06c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.370889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49595cef-3293-4bdb-b2e4-b188248f06c2 None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.371604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49595cef-3293-4bdb-b2e4-b188248f06c2 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:25.371948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49595cef-3293-4bdb-b2e4-b188248f06c2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.373395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49595cef-3293-4bdb-b2e4-b188248f06c2 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.390847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 230/470] 10.222.0.66 () {56 vars in 1139 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:25.391937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c34c9d-11d9-4a14-8090-f3111eb2a716 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.392328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c34c9d-11d9-4a14-8090-f3111eb2a716 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.392655 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c34c9d-11d9-4a14-8090-f3111eb2a716 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.394209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 231/471] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:25.396261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c34c9d-11d9-4a14-8090-f3111eb2a716 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.416324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9cdd62-fb6d-4eb4-93ec-ac9457fa9861 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.450047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.450675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.451019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.451299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.457974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.460147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.462064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:25.472286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:25.473172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:25.474046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 241/472] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:25.474498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.501699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c34c9d-11d9-4a14-8090-f3111eb2a716 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.536067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.536669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.536954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.537229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.538724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:25.538930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.540353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.551674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 232/473] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:25.552375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.580483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c54d41-c3e2-4086-bb71-d392815619d4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.587776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.613790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65769f6d-8c74-484c-9393-fd583464e188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.614314 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65769f6d-8c74-484c-9393-fd583464e188 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.614599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65769f6d-8c74-484c-9393-fd583464e188 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.614879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65769f6d-8c74-484c-9393-fd583464e188 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.653603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1419f3b-0f9d-4e51-83be-f4f20591c546 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.684842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.686892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:25.687505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.687790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.688064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.688738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65769f6d-8c74-484c-9393-fd583464e188 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.690238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65769f6d-8c74-484c-9393-fd583464e188 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.756732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:25.758209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:25.790850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.796073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 242/474] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:47:25.796073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b194849-1d22-436c-bc13-b3017f34aee9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:25.796073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b194849-1d22-436c-bc13-b3017f34aee9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:25.796073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b194849-1d22-436c-bc13-b3017f34aee9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:25.811529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1b194849-1d22-436c-bc13-b3017f34aee9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:25.812726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1b194849-1d22-436c-bc13-b3017f34aee9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:25.861969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 233/475] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 19:47:25.862812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.879963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 243/476] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:25.880746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b194849-1d22-436c-bc13-b3017f34aee9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:25.920005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:25.971703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6032e11d-555f-41e8-9a36-4dab1df813a1 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:25.995561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b194849-1d22-436c-bc13-b3017f34aee9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.022698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.023394 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.023683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.023956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.024652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:26.024968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.026455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.028998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:26.039694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 234/477] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:26.042559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.045082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.045732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.046018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.046368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.047035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] RBAC: Adding query filter params (role.id=57b3843b68f846d5905474ee762f7059, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=d54e8c38f0f84e4287b0445947bde916) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:26.047349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.048790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.055432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 244/478] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:25 2022] GET /identity/v3/role_assignments?user.id=d54e8c38f0f84e4287b0445947bde916&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=57b3843b68f846d5905474ee762f7059 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:26.062897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.172085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d434491-efc0-463a-8812-77981f9353fb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.185265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:26.196875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3d8184-926c-4e9e-b40f-aac4aeca7f01 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.208688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.210103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.210382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.210650 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.223344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.224815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.237254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.237929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.238195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.238468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:26.241421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:26.242428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:26.243292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:26.243292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:26.243292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:26.243292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 235/479] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:26.243292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.250849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.252298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:26.269774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:26.270718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:26.271538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:26.271538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:26.271538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:26.271538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 245/480] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:26.272342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.298172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:26.359025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e7d228-1254-4df2-98e4-a737bc577573 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.379857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691a5926-76f6-4aee-8225-fda4f7e08326 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.395019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.395635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.395921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.396200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.397484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] RBAC: Adding query filter params (role.id=57b3843b68f846d5905474ee762f7059, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=a7fae443ad92439aadd2f94ad3bca594) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:26.398401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.399864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.406134 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 236/481] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/role_assignments?user.id=a7fae443ad92439aadd2f94ad3bca594&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=57b3843b68f846d5905474ee762f7059 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:26.406818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.426093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.427104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.427583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.428085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.442616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:26.449946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:26.450880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:26.451690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:26.459507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 246/482] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:26.460834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.465394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:26.526675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae084ea7-7135-46a7-b733-9741ab8769c3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.539863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:26.567178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.567814 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.568102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.568382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.569834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:26.570169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.571613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.586108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 237/483] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:26.586829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.597466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342a6e5c-2995-47e3-915b-ab2ac354dac7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.640524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.641153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.641484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.641768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.642424 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:26.642733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.644131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.658188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 247/484] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:26.659830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.718571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7760e3-da22-4baf-bd0b-218ae1b591db None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.732662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:26.759351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.760032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.760320 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.760596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.761267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] RBAC: Adding query filter params (name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:26.761600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.762982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.772618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 238/485] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:26.773128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e1d66-5312-496b-98ba-1ef4cd5e21f1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.773582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.809280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:26.809494 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.810883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.811265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.811641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.819876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.821487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:26.833807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:26.835415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:26.836466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:26.838118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 248/486] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:26.838855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.886845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e789564-5bb3-4660-ae0e-703871e8f56c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.927027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:26.927661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:26.928159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:26.928945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:26.937995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:26.939905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:26.954663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:26.955746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:26.956936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 239/487] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:26.957552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:26.967856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6275fe90-0197-45e0-a1c2-e43b88cc31cb None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:26.993449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:27.005455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.006447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.007098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.007577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.020406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.022059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:27.042584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:27.043554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:27.044413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:27.044413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:27.044413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:27.048677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 249/488] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:27.049625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:27.080047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:27.092463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397dfb6e-6464-487b-a0b7-baedcfaf8be5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.136417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.137180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.137767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.138140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.139292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:27.139923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.141639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.157666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639d634e-a27d-420a-9543-bd052dc5421b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.161389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 240/489] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:27.162265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:27.205594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.206777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.207156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.207907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.208979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:27.209988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.212320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.230151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 250/490] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:27.233283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:27.271353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:27.278394 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2b390f-f4b2-44ca-a9b2-6bb4e1cc1cb2 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.316930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.317434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.317824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.318281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.320198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:27.320669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.322159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.338961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 241/491] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:27.338961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:27.348660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a267cd-11b5-4242-9743-2406a16f8b4e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.352165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:27.386592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.387448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.387954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.388400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.389269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] RBAC: Adding query filter params (role.id=57b3843b68f846d5905474ee762f7059, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=183a4f23782143b0b234d7800b409452) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:27.389746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.391330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.398631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 251/492] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:27 2022] GET /identity/v3/role_assignments?user.id=183a4f23782143b0b234d7800b409452&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=57b3843b68f846d5905474ee762f7059 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:27.399688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:27.478233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31620465-a8ea-4bcf-a782-1016582b3cdd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.509803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:27.513575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490028f4-e8fd-403f-bc36-df13ba966c02 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.542310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.543079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.543476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.543921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.544674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] RBAC: Adding query filter params (role.id=57b3843b68f846d5905474ee762f7059, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=91639dc527ae47e892164544572cbec4) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:27.545084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.546654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.552136 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.552999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.553592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.554000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.562207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 242/493] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:27 2022] GET /identity/v3/role_assignments?user.id=91639dc527ae47e892164544572cbec4&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=57b3843b68f846d5905474ee762f7059 => generated 608 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:27.563536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.565204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:27.579379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:27.580597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:27.581859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:27.584056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 252/494] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:27.590156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:27.684618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:27.710432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586b2b6-0a4b-452a-acf6-70206ae2bc3d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.748942 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.749767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.750164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.750536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.751368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:27.751773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:27.753347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:27.776118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 243/495] 10.222.0.66 () {56 vars in 1135 bytes} [Fri Oct 28 19:47:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:27.781750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:27.901966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e3630b-4167-42a5-a008-97796f627180 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:27.939958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:27.940850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:27.941277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:27.941722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:27.976068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:28.026419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:28.027981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:28.163332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 253/496] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 19:47:28.170593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:28.291420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8018bdb-f073-4fe5-aa83-02fdd989d1ba None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:28.328964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:28.330003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:28.330493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:28.331002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:28.333119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:28.343031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:28.353035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:28.361054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:28.362019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:28.362887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:28.362887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:28.362887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:28.362887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 244/497] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:28.368909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:28.438800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:28.492682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e9007d-3b8e-49d1-ba60-52819537871f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:28.528893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fc6afa-755f-49db-94bd-15443b1db76f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:28.529592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fc6afa-755f-49db-94bd-15443b1db76f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:28.529880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fc6afa-755f-49db-94bd-15443b1db76f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:28.530158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fc6afa-755f-49db-94bd-15443b1db76f None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:28.530803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fc6afa-755f-49db-94bd-15443b1db76f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:28.531108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fc6afa-755f-49db-94bd-15443b1db76f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:28.532552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fc6afa-755f-49db-94bd-15443b1db76f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:28.550794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 254/498] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:28.557265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:28.604046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7969ca4-08b8-4b02-8e89-bf72bacbda2e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:28.604357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7969ca4-08b8-4b02-8e89-bf72bacbda2e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:28.604630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7969ca4-08b8-4b02-8e89-bf72bacbda2e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:28.606058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 255/499] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:47:28.622816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976818fb-d585-41d2-bbca-8319b5ff2429 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:28.623119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976818fb-d585-41d2-bbca-8319b5ff2429 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:28.623382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976818fb-d585-41d2-bbca-8319b5ff2429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:28.650207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-976818fb-d585-41d2-bbca-8319b5ff2429 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:28.651426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-976818fb-d585-41d2-bbca-8319b5ff2429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:28.726171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabf9ae6-3aad-461f-855b-88268e74d109 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:28.768312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8fffde-3faf-4462-809d-d6c8e6ae194a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:28.768956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8fffde-3faf-4462-809d-d6c8e6ae194a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:28.769240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8fffde-3faf-4462-809d-d6c8e6ae194a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:28.769541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8fffde-3faf-4462-809d-d6c8e6ae194a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:28.779830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8fffde-3faf-4462-809d-d6c8e6ae194a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:28.781201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8fffde-3faf-4462-809d-d6c8e6ae194a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:28.783818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 256/500] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-5f8fffde-3faf-4462-809d-d6c8e6ae194a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:28.800909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:28.801906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:28.805042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 245/501] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:28.814716 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976818fb-d585-41d2-bbca-8319b5ff2429 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:28.905889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:28.966242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976818fb-d585-41d2-bbca-8319b5ff2429 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:29.020775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:29.021738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.022356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.022898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.023783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:29.024305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:29.026059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:29.041454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252cb41c-fd37-46ee-8ad3-435bebb9e805 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.041739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252cb41c-fd37-46ee-8ad3-435bebb9e805 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.042015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252cb41c-fd37-46ee-8ad3-435bebb9e805 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.042127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 257/502] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:29.043363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 246/503] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:29.047874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:29.053092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a00f33-7607-4a52-adef-7c0a27bb9647 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.054079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a00f33-7607-4a52-adef-7c0a27bb9647 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.054363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a00f33-7607-4a52-adef-7c0a27bb9647 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.081880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-95a00f33-7607-4a52-adef-7c0a27bb9647 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:29.083910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-95a00f33-7607-4a52-adef-7c0a27bb9647 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:29.148279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:29.173376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 247/504] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 19:47:29.174143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a00f33-7607-4a52-adef-7c0a27bb9647 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:29.221816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b18b04e-fa84-4cde-b09f-984151c929e8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:29.233871 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:29.265466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:29.266541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.267053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.267629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.268568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] RBAC: Adding query filter params (role.id=57b3843b68f846d5905474ee762f7059, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=0df958d5d645473c9c0403bf9b4ee846) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:29.269120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:29.270926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:29.283930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 258/505] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:29 2022] GET /identity/v3/role_assignments?user.id=0df958d5d645473c9c0403bf9b4ee846&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=57b3843b68f846d5905474ee762f7059 => generated 608 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:29.320504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a00f33-7607-4a52-adef-7c0a27bb9647 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:29.357769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:29.358412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.358700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.358980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.371022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:29.375995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:29.376931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:29.377818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 248/506] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:29.383083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:29.515940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46822d49-fcbc-4844-bebb-9acc51b4bebf None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:29.552668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:29.553918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.554259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.554537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.555204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:29.555512 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:29.556879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:29.565474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 259/507] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:29.571496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:29.621787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:29.629348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:29.680090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:29.734533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe8b294-0deb-41cf-a0f2-c0ac3224867c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:29.796132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:29.796780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.797060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.797345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.802593 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ca39e2-ee43-4039-b23f-f708af7fa10f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:29.821404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:29.823221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:29.849140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:29.850195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:29.851081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:29.851081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:29.851081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:29.852064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 249/508] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:29.858149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:29.874975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:29.875627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:29.876038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:29.876382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:29.903078 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:29.904709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:29.929337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:29.930401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:29.931282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:29.931282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:29.931282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:29.931282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 260/509] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:29.937162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:30.048520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:30.052577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:30.069633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56639ec0-d09e-411f-ac85-2c3a57e45f59 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:30.080219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a65608-2ea1-43c0-a454-348950b577f9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:30.103537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:30.104216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.104504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.104778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.105465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:30.105774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:30.107278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:30.120170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 250/510] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:30.125642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:30.167816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:30.168581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.168866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.169142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.169922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:30.170239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:30.172502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:30.202249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 261/511] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:30.207718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:30.253099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216fb6e6-1b49-4d4e-b524-9ace919aa95d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:30.292763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd36bed-b3d4-41c6-9623-45af3c53a3aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:30.293470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd36bed-b3d4-41c6-9623-45af3c53a3aa None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.293757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd36bed-b3d4-41c6-9623-45af3c53a3aa None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.294035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd36bed-b3d4-41c6-9623-45af3c53a3aa None admin] PATH_INFO: `/v3/users/nova` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.303631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd36bed-b3d4-41c6-9623-45af3c53a3aa None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:30.305358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd36bed-b3d4-41c6-9623-45af3c53a3aa None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-7cd36bed-b3d4-41c6-9623-45af3c53a3aa None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:30.320033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:30.321034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:30.322640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 251/512] 10.222.0.66 () {56 vars in 1093 bytes} [Fri Oct 28 19:47:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:30.323232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82bb12-4920-499e-8502-3490e9c68695 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.323232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82bb12-4920-499e-8502-3490e9c68695 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.323360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82bb12-4920-499e-8502-3490e9c68695 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.324821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 252/513] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:30.326748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a82bb12-4920-499e-8502-3490e9c68695 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:30.343304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:30.358867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70205419-9811-49ed-8876-f1014a604ffc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:30.395814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07e841-d8ef-42b8-b5fc-e1965bd2f00e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:30.396661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07e841-d8ef-42b8-b5fc-e1965bd2f00e None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.397182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07e841-d8ef-42b8-b5fc-e1965bd2f00e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.397695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07e841-d8ef-42b8-b5fc-e1965bd2f00e None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.398544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be07e841-d8ef-42b8-b5fc-e1965bd2f00e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:30.400197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be07e841-d8ef-42b8-b5fc-e1965bd2f00e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:30.448164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a82bb12-4920-499e-8502-3490e9c68695 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:30.481495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 262/514] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:47:30 2022] POST /identity/v3/users => generated 254 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:47:30.483228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af43df26-3156-44dc-bbb3-df4c3b36161d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.483523 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af43df26-3156-44dc-bbb3-df4c3b36161d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.483801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af43df26-3156-44dc-bbb3-df4c3b36161d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.493981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f0445-9e56-4dd2-8162-ff78ea33d117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:30.494450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:30.494623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f0445-9e56-4dd2-8162-ff78ea33d117 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.494882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f0445-9e56-4dd2-8162-ff78ea33d117 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.495200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f0445-9e56-4dd2-8162-ff78ea33d117 None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.496075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f0445-9e56-4dd2-8162-ff78ea33d117 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:30.496272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f0445-9e56-4dd2-8162-ff78ea33d117 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:30.504129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f0445-9e56-4dd2-8162-ff78ea33d117 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:30.512019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-af43df26-3156-44dc-bbb3-df4c3b36161d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:30.516439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-af43df26-3156-44dc-bbb3-df4c3b36161d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:30.529522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 253/515] 10.222.0.66 () {56 vars in 1133 bytes} [Fri Oct 28 19:47:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:30.531836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395561c-35e9-48e8-8ae6-cdd1e796a0fb None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.532125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395561c-35e9-48e8-8ae6-cdd1e796a0fb None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.532402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395561c-35e9-48e8-8ae6-cdd1e796a0fb None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.539093 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 254/516] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:30.545222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c395561c-35e9-48e8-8ae6-cdd1e796a0fb None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:30.596510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 263/517] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:30.597893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19ca7a8-6585-4e2c-8286-bbe3c8b18272 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.598280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19ca7a8-6585-4e2c-8286-bbe3c8b18272 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.598658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19ca7a8-6585-4e2c-8286-bbe3c8b18272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.623693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f19ca7a8-6585-4e2c-8286-bbe3c8b18272 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:30.624897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f19ca7a8-6585-4e2c-8286-bbe3c8b18272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:30.682992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:30.733661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 264/518] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:30.736018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c395561c-35e9-48e8-8ae6-cdd1e796a0fb None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:30.771500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:30.772557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.773077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.773579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.784638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:30.786733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:30.803766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:30.804740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:30.805678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:30.805678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:30.805678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:30.807769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 255/519] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:30.815758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19ca7a8-6585-4e2c-8286-bbe3c8b18272 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:30.887715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:30.915523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:30.930090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19ca7a8-6585-4e2c-8286-bbe3c8b18272 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:30.981626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:30.982253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:30.982540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:30.982819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:30.983471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:30.983779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:30.985157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.015294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 265/520] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:31.016577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.069954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c87880-9cd9-4e4d-943b-fd38b9a4e74e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.092103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:31.113127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.114287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.114636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.114980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] PATH_INFO: `/v3/services/network` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.116432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.118282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:31.133063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:31.134125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:31.134985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:31.135726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 256/521] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:30 2022] GET /identity/v3/services/network => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:31.136416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.145105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7edeb4a-c875-4c11-8df6-d20e0e38719c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.171790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:31.184995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.186237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.186792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.187222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.197594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.199629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:31.216277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:31.217234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 266/522] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:31.221918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.263989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9bfdf9-de29-4714-bc8b-f6f186d2eeeb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.306145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.307117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.307467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.307806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.309309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.311140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:31.324000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:31.325243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:31.326209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:31.329470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 257/523] 10.222.0.66 () {56 vars in 1129 bytes} [Fri Oct 28 19:47:31 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:31.330398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.349533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a883e8-d3df-49e3-babb-622abbbb98c3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.391506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:31.395626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.396194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.396482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.396775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.398433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] RBAC: Adding query filter params (name=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:31.398543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.403234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.413653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 267/524] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:31 2022] GET /identity/v3/services?name=network => generated 124 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:31.414566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.476587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d95176-f88f-461a-8f59-2a54d6ae4efd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.486524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:31.526313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.527268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.527867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.528326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.529487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:31.530006 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.532086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.549600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9be4cf6-ae55-4079-9cf8-db19b3c13380 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.553142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 258/525] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:31.553926 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.599483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.602063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.602649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.603116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.603934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:31.604782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.607252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.613256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 268/526] 10.222.0.66 () {56 vars in 1124 bytes} [Fri Oct 28 19:47:31 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:31.614349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.692820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc14d0-a27b-47cc-98ec-f28af16f104c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.703426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:31.734659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.735370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.735568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.735843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.736484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] RBAC: Adding query filter params (type=network) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:31.736789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.738407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.751864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 259/527] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:31 2022] GET /identity/v3/services?type=network => generated 124 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:31.751864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:31.752126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf1b774-182d-421f-a3d2-1ee6546d95e0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.778228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:31.823592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d12a1-a048-4d37-93f5-d8636d2b251b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.824348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d12a1-a048-4d37-93f5-d8636d2b251b None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.825748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d12a1-a048-4d37-93f5-d8636d2b251b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.825748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d12a1-a048-4d37-93f5-d8636d2b251b None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/d54e8c38f0f84e4287b0445947bde916/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.829299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82d12a1-a048-4d37-93f5-d8636d2b251b None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=d54e8c38f0f84e4287b0445947bde916, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.833137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82d12a1-a048-4d37-93f5-d8636d2b251b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.895159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 269/528] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:31 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/d54e8c38f0f84e4287b0445947bde916/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:31.897633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c550c-5588-4269-a376-019328042889 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.898156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c550c-5588-4269-a376-019328042889 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.898675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c550c-5588-4269-a376-019328042889 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.905961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 270/529] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:31 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:31.908059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47991a83-609b-498a-9a0d-ba420973a1ff None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.908594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47991a83-609b-498a-9a0d-ba420973a1ff None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.909103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47991a83-609b-498a-9a0d-ba420973a1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.926948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd6dcf8-374d-4069-b304-8f0597387dc4 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:31.936308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-47991a83-609b-498a-9a0d-ba420973a1ff None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:31.938000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-47991a83-609b-498a-9a0d-ba420973a1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:31.978017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:31.978566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:31.978898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:31.979172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:31.979950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:31.980253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:31.981745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:31.987455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 260/530] 10.222.0.66 () {56 vars in 1124 bytes} [Fri Oct 28 19:47:31 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:32.052886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 271/531] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:32.334948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:32.372723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:32.444086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd61f37-2be8-451b-9d00-272f3c59cde4 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:32.481289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:32.482333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:32.482821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:32.483297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:32.495424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:32.501237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:32.502183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:32.503027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:32.505652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 261/532] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:32.511721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47991a83-609b-498a-9a0d-ba420973a1ff None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:32.618704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47991a83-609b-498a-9a0d-ba420973a1ff None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:32.620680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:32.654887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:32.655948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:32.656437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:32.656919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:32.657856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:32.658382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:32.659948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:32.669361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 272/533] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:32.675168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:32.776001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:32.784016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895eaaa2-0c79-434a-8826-ab7164269faf None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:32.819619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:32.820899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:32.821568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:32.822088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:32.833616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:32.836213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:32.851884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:32.852903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:32.853807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:32.853807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:32.853807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:32.855930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 262/534] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:32.862007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:32.970120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55e360a-d424-498c-beae-31968faaea99 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:33.005680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:33.007279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:33.007881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:33.008361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:33.014503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:33.015103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:33.016770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:33.036329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 273/535] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:33.042564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:33.155012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b56179c-bcbd-494d-8035-b4eb6861057e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:33.193100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be79c402-8a48-4f0c-b7c3-3953ff3009b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:33.194249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be79c402-8a48-4f0c-b7c3-3953ff3009b0 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:33.194742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be79c402-8a48-4f0c-b7c3-3953ff3009b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:33.195224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be79c402-8a48-4f0c-b7c3-3953ff3009b0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:33.203255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be79c402-8a48-4f0c-b7c3-3953ff3009b0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:33.204945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be79c402-8a48-4f0c-b7c3-3953ff3009b0 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-be79c402-8a48-4f0c-b7c3-3953ff3009b0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:33.216987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:33.217935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:33.218878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:33.221050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 263/536] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:33.227256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:33.253732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:33.333509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c893314e-af86-4f53-9ad2-71c4b0cd834f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:33.369706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:33.370320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:33.370607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:33.370884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:33.371614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:33.371924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:33.373433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:33.386672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeaf92b-3b4e-4e40-8dee-c85b8cecca08 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:33.386961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeaf92b-3b4e-4e40-8dee-c85b8cecca08 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:33.387239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeaf92b-3b4e-4e40-8dee-c85b8cecca08 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:33.388861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 264/537] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:47:33.394054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1d13e1-7fc6-41d6-8918-b3f533ce5f00 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:33.394340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1d13e1-7fc6-41d6-8918-b3f533ce5f00 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:33.394620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1d13e1-7fc6-41d6-8918-b3f533ce5f00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:33.409691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:33.412099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 274/538] 10.222.0.66 () {56 vars in 1135 bytes} [Fri Oct 28 19:47:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:33.421779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-9d1d13e1-7fc6-41d6-8918-b3f533ce5f00 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:33.422983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9d1d13e1-7fc6-41d6-8918-b3f533ce5f00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:33.438694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:33.524371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 265/539] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:33.578866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:33.606147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc4230a-85bb-48c1-aa5f-0ded96e5b810 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:33.659130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:33.659783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:33.660069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:33.660348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:33.674305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:33.675757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:33.695683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:33.696633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:33.697529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:33.697529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:33.697529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:33.701403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 275/540] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:33.702781 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1d13e1-7fc6-41d6-8918-b3f533ce5f00 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:33.845787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1d13e1-7fc6-41d6-8918-b3f533ce5f00 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:33.894860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:33.895840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:33.896330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:33.897408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:33.898268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:33.898771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:33.900564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:33.910751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:33.916568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 266/541] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:33.922200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:34.000120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:34.103634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b127e4c-33fd-429c-be3d-8b210e0a0171 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:34.174614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:34.175260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.175544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.175820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.179574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:34.191101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:34.192590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:34.214974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:34.215952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:34.216860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:34.220157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e55e37-965a-44de-919c-6c271f367722 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:34.228562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 276/542] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:34.230123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:34.274372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:34.275013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.275297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.275572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.287181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:34.293189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:34.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:34.295391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 267/543] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:34.302666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:34.422798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:34.431065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1313a2-0464-4fe0-9977-a8d7a9313bdc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:34.509124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:34.509806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.510092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.510370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.514219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:34.514540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:34.516058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:34.521185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0afc25e-d291-432d-869a-6037cd1d4b5d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:34.548652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 277/544] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:34.554811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:34.562803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:34.563437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.563842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.564126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.564806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] RBAC: Adding query filter params (name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:34.565185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:34.566745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:34.577606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 268/545] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:34.583000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:34.667017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:34.698880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e63a061-49e5-4c5c-9afb-e0120cc9dd4a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:34.713762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd67e6-8541-4061-92cc-05fe257cc789 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:34.715850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:34.731861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197b9ad9-8081-40b5-98be-43d582ff5db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:34.732490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b9ad9-8081-40b5-98be-43d582ff5db3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.732772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b9ad9-8081-40b5-98be-43d582ff5db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.733045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b9ad9-8081-40b5-98be-43d582ff5db3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.744137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197b9ad9-8081-40b5-98be-43d582ff5db3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:34.745628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197b9ad9-8081-40b5-98be-43d582ff5db3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:34.747347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799d892-2f1b-445c-8254-be1a423c239f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:34.748011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799d892-2f1b-445c-8254-be1a423c239f None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.748298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799d892-2f1b-445c-8254-be1a423c239f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.748583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799d892-2f1b-445c-8254-be1a423c239f None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/0df958d5d645473c9c0403bf9b4ee846/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.750461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7799d892-2f1b-445c-8254-be1a423c239f None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=0df958d5d645473c9c0403bf9b4ee846, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:34.751876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7799d892-2f1b-445c-8254-be1a423c239f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-197b9ad9-8081-40b5-98be-43d582ff5db3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:34.760803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:34.762043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:34.763228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:34.763228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:34.763228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:34.763228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 269/546] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:34.763228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dfc8ae-12f7-4bb6-90b9-2125bb81ec71 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.763588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dfc8ae-12f7-4bb6-90b9-2125bb81ec71 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.763588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dfc8ae-12f7-4bb6-90b9-2125bb81ec71 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.764734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 270/547] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:34.765756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c06b3-ca88-425b-a3a6-9c5cc33aa5d5 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.766071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c06b3-ca88-425b-a3a6-9c5cc33aa5d5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.766343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c06b3-ca88-425b-a3a6-9c5cc33aa5d5 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.767491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 271/548] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:34.768489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda90ce-ebdf-4edd-9a8a-b88920e20ae1 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.768767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda90ce-ebdf-4edd-9a8a-b88920e20ae1 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.769115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda90ce-ebdf-4edd-9a8a-b88920e20ae1 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.770219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 272/549] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:34.770895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda90ce-ebdf-4edd-9a8a-b88920e20ae1 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:34.822831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 278/550] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:34 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/0df958d5d645473c9c0403bf9b4ee846/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:47:34.824459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4faa64b-2195-475a-8a0d-02725e968360 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.824752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4faa64b-2195-475a-8a0d-02725e968360 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.825073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4faa64b-2195-475a-8a0d-02725e968360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.851296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c4faa64b-2195-475a-8a0d-02725e968360 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:34.852541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c4faa64b-2195-475a-8a0d-02725e968360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:34.880307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda90ce-ebdf-4edd-9a8a-b88920e20ae1 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:34.918907 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07314dcc-722c-4bed-a226-cf1da83455cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:34.919528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07314dcc-722c-4bed-a226-cf1da83455cf None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.919809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07314dcc-722c-4bed-a226-cf1da83455cf None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.920091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07314dcc-722c-4bed-a226-cf1da83455cf None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.920730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07314dcc-722c-4bed-a226-cf1da83455cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:34.921030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07314dcc-722c-4bed-a226-cf1da83455cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:34.922602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07314dcc-722c-4bed-a226-cf1da83455cf None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:34.934087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 279/551] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:34.935427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ea792-89a4-4f61-ba67-c33f47465aa5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.935719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ea792-89a4-4f61-ba67-c33f47465aa5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.936272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ea792-89a4-4f61-ba67-c33f47465aa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.941661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 273/552] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:34.941661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c8b39-08a5-4b2c-92ce-1f4850715804 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:34.941661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c8b39-08a5-4b2c-92ce-1f4850715804 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:34.941661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c8b39-08a5-4b2c-92ce-1f4850715804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:34.960162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1e0ea792-89a4-4f61-ba67-c33f47465aa5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:34.961364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1e0ea792-89a4-4f61-ba67-c33f47465aa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:34.966920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-1d5c8b39-08a5-4b2c-92ce-1f4850715804 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:34.968513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1d5c8b39-08a5-4b2c-92ce-1f4850715804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:35.000935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:35.007182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:35.043796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 280/553] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:35.045398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ea792-89a4-4f61-ba67-c33f47465aa5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:35.082623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 274/554] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 19:47:35.151742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ea792-89a4-4f61-ba67-c33f47465aa5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:35.186575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:35.187205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:35.187483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:35.187755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:35.195695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:35.197192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:35.208934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:35.210856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:35.211814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 281/555] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:47:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:35.218611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5c8b39-08a5-4b2c-92ce-1f4850715804 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:35.235718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:35.290439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:35.383524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5c8b39-08a5-4b2c-92ce-1f4850715804 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:35.388208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453a575d-0e6a-42b2-bbbd-4b4ff49edf4a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:35.422203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:35.423691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:35.423691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:35.423691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:35.424060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:35.424889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:35.425794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:35.433748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:35.434611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:35.434891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:35.435164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:35.450704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:35.450877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 275/556] 10.222.0.66 () {56 vars in 1149 bytes} [Fri Oct 28 19:47:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:35.450877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:35.451948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:35.452865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 282/557] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:35.454372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:35.473889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:35.551735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:35.559551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7516c1b-0b80-4b7d-8128-cae6f845aad3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:35.565517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7551e0-6ed0-4454-b29d-67059e7d070b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:35.597484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:35.598091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:35.598370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:35.598634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:35.599274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:35.600800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:35.600906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:35.601450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:35.601738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:35.602011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:35.602643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:35.604008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:35.686333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 283/558] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:47:35 2022] POST /identity/v3/services => generated 240 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:35.687165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:35.715187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 276/559] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:47:35 2022] POST /identity/v3/services => generated 251 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:35.716393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:35.901477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:35.936990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60c8ddc-11a7-4cfc-8974-a9b99da96ead None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:35.943053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878f0a1-6808-4302-b4b5-b8172bce316c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:35.987454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:35.987887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:35.988511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:35.988793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:35.989067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:35.989350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:35.989754 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:35.990059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:35.990336 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:35.990966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:35.991539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.000847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 277/560] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:36.004199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.005368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.007588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:36.020929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:36.022185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:36.023065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:36.023065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:36.023065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:36.023065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 284/561] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:36.031932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.034868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:36.134837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0531acf4-20d3-488c-84d3-df7fa94c8e95 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:36.160137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b701962-9584-4e2c-8821-0fb04bdf3e9c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:36.182354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:36.183834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:36.184307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:36.185022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:36.189368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:36.200791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.202792 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.210174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:36.210927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:36.211283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:36.211630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:36.212691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:36.212691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.214159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:36.219822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:36.220738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:36.221666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:36.221666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:36.221666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:36.224105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 278/562] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:36.230263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.239501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 285/563] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:36.245606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.370784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:36.391524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c017832-001e-4072-925a-2b53fd101a52 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:36.447627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c354e2-eec7-453f-837a-4dfc2fd07bb2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:36.470261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:36.470876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:36.471152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:36.471482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:36.472119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:36.472417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.473817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.492410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 279/564] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:36.497474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.527210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:36.528692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:36.529816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:36.531714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:36.546787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.549843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:36.580050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:36.580998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:36.581940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:36.587090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 286/565] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:36.592742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.606199 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:36.712245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e19c47-fd9f-4878-8962-03a333129849 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:36.745775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:36.753969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:36.755024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:36.755531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:36.756039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:36.756951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:36.757631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.759350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.773920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 287/566] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:36.775919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9083927-c93d-436e-b0fb-8297afbd669e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:36.785696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.882974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:36.883621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:36.883896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:36.884167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:36.901507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.902970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:36.916012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88650c3d-bcf3-4f9d-832c-388bced8dbfe None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:36.933694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:36.934702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:47:36.935665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:36.938272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 280/567] 10.222.0.66 () {56 vars in 1093 bytes} [Fri Oct 28 19:47:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 441 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:36.943912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:36.950334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:36.951000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:36.951282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:36.951549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/2697ebcd1ef3492799fb7d4433b9f00f/roles/57b3843b68f846d5905474ee762f7059` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:36.967344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=2697ebcd1ef3492799fb7d4433b9f00f, role_id=57b3843b68f846d5905474ee762f7059)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:36.968777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:37.065239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8eab90-6c10-4142-aecb-f2b49e130fe9 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:37.080138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 288/568] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:36 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/2697ebcd1ef3492799fb7d4433b9f00f/roles/57b3843b68f846d5905474ee762f7059 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:37.084643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:37.113985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8616ba7f-fd1c-4716-94fd-e6f7f534db1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:37.114717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8616ba7f-fd1c-4716-94fd-e6f7f534db1b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:37.115010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8616ba7f-fd1c-4716-94fd-e6f7f534db1b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:37.115288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8616ba7f-fd1c-4716-94fd-e6f7f534db1b None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:37.122393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8616ba7f-fd1c-4716-94fd-e6f7f534db1b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:37.122710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8616ba7f-fd1c-4716-94fd-e6f7f534db1b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:37.124266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8616ba7f-fd1c-4716-94fd-e6f7f534db1b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:37.145687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 281/569] 10.222.0.66 () {56 vars in 1133 bytes} [Fri Oct 28 19:47:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:37.151997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:37.200493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:37.302592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1666620-f509-475d-9a57-9c714e1f1a09 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:37.343848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:37.344469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:37.344756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:37.345030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:37.417208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:37.417564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995efaa9-ffe4-46f8-9d79-607796fa42ea None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:37.418031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995efaa9-ffe4-46f8-9d79-607796fa42ea None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:37.418500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995efaa9-ffe4-46f8-9d79-607796fa42ea None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:37.418775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:37.420518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 282/570] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:47:37.425974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bbbe6e-2d07-405b-af4a-c7b1817a7006 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:37.426504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bbbe6e-2d07-405b-af4a-c7b1817a7006 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:37.426982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bbbe6e-2d07-405b-af4a-c7b1817a7006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:37.447301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-11bbbe6e-2d07-405b-af4a-c7b1817a7006 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:37.448912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-11bbbe6e-2d07-405b-af4a-c7b1817a7006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:37.453701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:37.535192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 283/571] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:37.561170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 289/572] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:37 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 19:47:37.566960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bbbe6e-2d07-405b-af4a-c7b1817a7006 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:37.678495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:37.682255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bbbe6e-2d07-405b-af4a-c7b1817a7006 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:37.733401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:37.734101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:37.734384 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:37.734656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:37.748895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:37.750668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:37.774166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:37.775209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:37.776095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:37.776095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:37.776095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:37.778253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 284/573] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:37.783792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:37.895184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:37.975219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d151426-c010-4889-b395-424b795c1c2a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.042393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.043282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.043762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.044169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.044969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:38.045442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:38.046967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:38.053691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:38.073123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 290/574] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:38.079533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:38.086934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d40d78-033e-49ba-83f1-a09f6ac7c6be None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.159573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.160456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.160888 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.161344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.194438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:38.199825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:38.200770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:38.201626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 285/575] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:38.210323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:38.224224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4828f9d-608f-42b8-bd56-655daf366b35 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.239803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:38.260815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c22540-30d0-46f5-ad6d-f14591385dda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.261492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22540-30d0-46f5-ad6d-f14591385dda None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.261776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22540-30d0-46f5-ad6d-f14591385dda None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.262050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22540-30d0-46f5-ad6d-f14591385dda None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.268860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c22540-30d0-46f5-ad6d-f14591385dda None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:38.270307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c22540-30d0-46f5-ad6d-f14591385dda None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-68c22540-30d0-46f5-ad6d-f14591385dda None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:38.283100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:38.284194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:38.285261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:38.289393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 291/576] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:38.290817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64886978-92e1-4e4b-a427-6154bcde5175 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.291213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64886978-92e1-4e4b-a427-6154bcde5175 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.291352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64886978-92e1-4e4b-a427-6154bcde5175 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.298221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 292/577] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:38.298221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64886978-92e1-4e4b-a427-6154bcde5175 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:38.357772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:38.366549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b9dfe1-d342-4764-8961-1c54fc25368d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.403456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcefa37-359b-4f7e-abb6-91c451d1b539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.404130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcefa37-359b-4f7e-abb6-91c451d1b539 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.404413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcefa37-359b-4f7e-abb6-91c451d1b539 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.404705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcefa37-359b-4f7e-abb6-91c451d1b539 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.405499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcefa37-359b-4f7e-abb6-91c451d1b539 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:38.405875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcefa37-359b-4f7e-abb6-91c451d1b539 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:38.407243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcefa37-359b-4f7e-abb6-91c451d1b539 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:38.409489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64886978-92e1-4e4b-a427-6154bcde5175 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.416752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 286/578] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:38.418037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af8a963-329c-4223-8a9a-1b74213a7529 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.418324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af8a963-329c-4223-8a9a-1b74213a7529 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.418600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af8a963-329c-4223-8a9a-1b74213a7529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.437571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8af8a963-329c-4223-8a9a-1b74213a7529 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:38.438743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8af8a963-329c-4223-8a9a-1b74213a7529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:38.445248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.445894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.446173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.446545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.447237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:38.447538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:38.449051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:38.462515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 293/579] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:38.468100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:38.506137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:38.511297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 287/580] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:38 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:38.512392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af8a963-329c-4223-8a9a-1b74213a7529 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:38.617090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af8a963-329c-4223-8a9a-1b74213a7529 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.627248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661ce81-327a-4569-a48b-02eefc57b9a0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.652200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.652811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.653095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.653395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.653820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:38.654127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=d54e8c38f0f84e4287b0445947bde916) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:38.654448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:38.656396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:38.663858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 288/581] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/role_assignments?user.id=d54e8c38f0f84e4287b0445947bde916&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:38.682030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee10234-2fc0-4186-9ac7-f347c607787f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.682613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee10234-2fc0-4186-9ac7-f347c607787f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.682900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee10234-2fc0-4186-9ac7-f347c607787f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.683178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee10234-2fc0-4186-9ac7-f347c607787f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.695848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee10234-2fc0-4186-9ac7-f347c607787f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:38.697392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee10234-2fc0-4186-9ac7-f347c607787f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-eee10234-2fc0-4186-9ac7-f347c607787f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:38.713220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:38.714288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:38.715644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:38.715644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:38.715644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:38.715644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 294/582] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:38.720177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:38.794809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9e7004-b666-446a-8e3e-96ba3d6da2ef None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.795107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9e7004-b666-446a-8e3e-96ba3d6da2ef None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.805201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9e7004-b666-446a-8e3e-96ba3d6da2ef None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.805201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 295/583] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:38.805201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaccfbe8-8228-451e-97ff-748820219215 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.805201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaccfbe8-8228-451e-97ff-748820219215 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.805201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaccfbe8-8228-451e-97ff-748820219215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.831911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:38.854508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-aaccfbe8-8228-451e-97ff-748820219215 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:38.855763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aaccfbe8-8228-451e-97ff-748820219215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:38.866159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e554ff68-45a6-4fa2-825b-79121bb54cfb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:38.906361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:38.906973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:38.907250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:38.907561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:38.908562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:38.908562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:38.909898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:38.925804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 289/584] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:38.926651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:38.948774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 296/585] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:38 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:38.949810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaccfbe8-8228-451e-97ff-748820219215 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.050412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.057754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaccfbe8-8228-451e-97ff-748820219215 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.059166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ebf705-df7b-43be-8215-ed456a966c90 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.084815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.093950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.094696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.094981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.095277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.097520 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.098722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.099222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.099856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] PATH_INFO: `/v3/services/network` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.100644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.102586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.103031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.104653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:39.112622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:39.122207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:39.123071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 290/586] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/services/network => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:39.123946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:39.124767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:39.129906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 297/587] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:47:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:39.131605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.132082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.238981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d46afd-07c7-49fa-8d4b-85d1eba489a5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.277478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb5a36a-e498-4381-9eaa-a2b2a83c78c7 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.300457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.301107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.301441 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.301718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.302549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:39.303406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.304975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.318055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.319226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.319745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.320215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.321102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] RBAC: Adding query filter params (name=network) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:39.321641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.323254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.324583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 298/588] 10.222.0.66 () {56 vars in 1135 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:39.325676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.328941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 291/589] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/services?name=network => generated 124 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:39.330013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.341955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.384325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.431388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afbfb98-2c37-4912-96a1-a97503badf6e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.433218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b7c9cf-8406-43fa-84fa-30ea25584b30 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.471818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.472553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.472834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.473106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.473742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.475709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.476400 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561e458b-b6e1-45e8-b1fb-addd90042aab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.477140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e458b-b6e1-45e8-b1fb-addd90042aab None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.477539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e458b-b6e1-45e8-b1fb-addd90042aab None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.477971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e458b-b6e1-45e8-b1fb-addd90042aab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:39.490513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:39.491634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:39.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:39.497365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 299/590] 10.222.0.66 () {56 vars in 1129 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:39.498251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.554691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561e458b-b6e1-45e8-b1fb-addd90042aab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.556318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561e458b-b6e1-45e8-b1fb-addd90042aab None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.583650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.609655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603a30c-cf61-447f-bf6f-ca7f6b71d881 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.614389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.648847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.649535 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.649818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.650095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.650717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] RBAC: Adding query filter params (type=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:39.651017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.652390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.658599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 300/591] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/services?type=network => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:39.659390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.675177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 292/592] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 19:47:39.676899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee19f62-3e3a-4841-8313-bd55e9e00925 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.677256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee19f62-3e3a-4841-8313-bd55e9e00925 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.677624 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee19f62-3e3a-4841-8313-bd55e9e00925 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.679963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 293/593] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:39.680786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee19f62-3e3a-4841-8313-bd55e9e00925 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.770936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d593a8-73d2-406a-9916-c3afb688aa35 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.789088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee19f62-3e3a-4841-8313-bd55e9e00925 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.813122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.813767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.814049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.814323 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.814950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:39.815252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.816742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.822215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 301/594] 10.222.0.66 () {56 vars in 1124 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:39.823036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.829885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.832272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87038ef2-b27c-433a-9ccb-003a16aeed22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.832948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87038ef2-b27c-433a-9ccb-003a16aeed22 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.833232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87038ef2-b27c-433a-9ccb-003a16aeed22 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.833537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87038ef2-b27c-433a-9ccb-003a16aeed22 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.834912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87038ef2-b27c-433a-9ccb-003a16aeed22 None admin] RBAC: Adding query filter params (interface=public, service_id=07236eda207b45ad8b2bac261553a6fb, region_id=RegionOne) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:39.835326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87038ef2-b27c-433a-9ccb-003a16aeed22 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.839433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87038ef2-b27c-433a-9ccb-003a16aeed22 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:39.844870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 294/595] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/endpoints?service_id=07236eda207b45ad8b2bac261553a6fb&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:39.847091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d34c691-77e0-430e-8be8-cb275dc05e9e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.847593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d34c691-77e0-430e-8be8-cb275dc05e9e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.848061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d34c691-77e0-430e-8be8-cb275dc05e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.869357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0d34c691-77e0-430e-8be8-cb275dc05e9e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:39.870596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0d34c691-77e0-430e-8be8-cb275dc05e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:39.885178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:39.932769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc221b53-7024-424f-9c34-441cf7e41562 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:39.947670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 295/596] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:39 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:39.948752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d34c691-77e0-430e-8be8-cb275dc05e9e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:39.974048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:39.974926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:39.975217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:39.975519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:39.986821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:39.988328 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:40.003865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:40.004865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:40.005816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:40.005816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:40.005816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:40.005816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 302/597] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:40.005816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.085691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d34c691-77e0-430e-8be8-cb275dc05e9e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.088043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:40.123319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.125360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.125360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.125360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.125360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:40.125827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.127383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.130813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d964c02-e514-4ffe-a99e-0d2dfa421e29 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.133914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 296/598] 10.222.0.66 () {56 vars in 1124 bytes} [Fri Oct 28 19:47:39 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:40.134290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.147408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:40.167596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.168527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.168989 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.169555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.170380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.171965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.185670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 303/599] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/services => generated 772 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:40.187243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.294346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4106e43-11b1-4e68-8247-a7d045c48b9d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.345143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.345831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.346113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.346387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.347009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dcc9f1-43fc-4101-a3f7-1b060b1290e6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.347120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:40.347329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.348833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.362101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 297/600] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:40.363040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.371673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:40.388185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.391881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.391881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.391881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.391881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] RBAC: Adding query filter params (interface=public, service_id=27bb0cab423346ab95a0976424817362, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:40.391881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.393696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.402869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 304/601] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/endpoints?service_id=27bb0cab423346ab95a0976424817362&interface=public®ion_id=RegionOne => generated 194 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:40.403710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.447372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:40.479502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef7d273-8593-4b0a-96d4-b41553d1c182 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.515809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.516785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.517262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.517756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.528489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:40.531173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cae4839-a959-4985-87d4-0e26c9c1d1e1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:40.535261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:40.536175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:47:40.537001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:40.540417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 298/602] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:40.541478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.577599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.578232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.578521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.578802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.587994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.590390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:40.606748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:40.607678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:40.608563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:40.612953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 305/603] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:40.613844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.654307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d31374-acf9-4173-b020-9ee74627abe0 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.676078 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:40.686537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:40.692836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.695997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.695997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.695997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.695997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.697160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.703014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 299/604] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/services => generated 772 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:40.704129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.741489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c538b-6a3f-4582-b905-e171d6f11fd3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.782332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.782947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.783231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.783508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.784235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:40.784469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.786217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.807630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 306/605] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:40.813427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.823204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c02791-a30b-4f79-b155-7a8761d9b3e0 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:40.862641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:40.863634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:40.864118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:40.864596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:40.866304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:40.867197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:40.869887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:40.885088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 300/606] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:40.891694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:40.954222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:41.011758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279fca4a-c6dd-4f15-9ebd-a586f7a02e18 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:41.014820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5415ea-0160-41b7-b3a5-c6ef5aae21cb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:41.025118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:41.081597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd82f9e-1ff4-4822-a555-4c07ee5a84cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:41.082264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd82f9e-1ff4-4822-a555-4c07ee5a84cd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.082546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd82f9e-1ff4-4822-a555-4c07ee5a84cd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.082824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd82f9e-1ff4-4822-a555-4c07ee5a84cd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.088820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:41.089497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.089776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.090046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.090797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=0df958d5d645473c9c0403bf9b4ee846) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:41.091104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:41.093723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:41.109286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 301/607] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/role_assignments?user.id=0df958d5d645473c9c0403bf9b4ee846&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:41.114503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd82f9e-1ff4-4822-a555-4c07ee5a84cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:41.116546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd82f9e-1ff4-4822-a555-4c07ee5a84cd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-abd82f9e-1ff4-4822-a555-4c07ee5a84cd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:41.150522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:41.151620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:41.152505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:41.152505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:41.152505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:41.152505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 307/608] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:41.157596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:41.225509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0cbea2-a144-4975-a9e5-33df73307e5c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.225840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0cbea2-a144-4975-a9e5-33df73307e5c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.226228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0cbea2-a144-4975-a9e5-33df73307e5c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.227908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 308/609] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:41.233249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b88dfdd-bd30-42b1-8dee-0f8eb3497705 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.233581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b88dfdd-bd30-42b1-8dee-0f8eb3497705 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.233862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b88dfdd-bd30-42b1-8dee-0f8eb3497705 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.254864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7b88dfdd-bd30-42b1-8dee-0f8eb3497705 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:41.256969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7b88dfdd-bd30-42b1-8dee-0f8eb3497705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:41.328024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4bec4-b8f8-4665-8a5f-cfc0b6c3e4d5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:41.349557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:41.375088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:41.375676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.375956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.376524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.377733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:41.378047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:41.379679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:41.384913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 309/610] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:41 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:41.397423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 302/611] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:41.399325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b88dfdd-bd30-42b1-8dee-0f8eb3497705 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:41.491642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:41.504885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b88dfdd-bd30-42b1-8dee-0f8eb3497705 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:41.541597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:41.542572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.543036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.543484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.551452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:41.553830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:41.566237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:41.567190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:41.568129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:41.574414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 310/612] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:47:41 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:41.576734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:41.680554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b52469-ad4a-4021-a5e2-f613fceffb64 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:41.716169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:41.717051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.717503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.717916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.718775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:41.719239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:41.720818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:41.745584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 303/613] 10.222.0.66 () {56 vars in 1149 bytes} [Fri Oct 28 19:47:41 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:41.747219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:41.750491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:41.835393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:41.868956 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c55847-9bd8-4c82-a4f5-b82e8c01e613 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:41.906674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:41.907437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:41.907819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:41.908192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:41.932283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:42.000756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142f4536-cdc0-4c9e-b1dd-f5ec0a817c8f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:42.012917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:42.014518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:42.041728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:42.042492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.042875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.043250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:42.043930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:42.045494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:42.056185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:42.057403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:42.058569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:42.060679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 304/614] 10.222.0.66 () {56 vars in 1132 bytes} [Fri Oct 28 19:47:41 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:42.068293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:42.184255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc8980-6b99-47c8-8e9e-fbdd5880673a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:42.217314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:42.223123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:42.223835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.224176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.224517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:42.225860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:42.226231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:42.228230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:42.233350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 305/615] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:42 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:42.238676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:42.308205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 311/616] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:41 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 561 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 19:47:42.322751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:42.364261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7a0d7-2dd4-45c7-a341-f97a076a64d8 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:42.405027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:42.405667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.405948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.406223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:42.409341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:42.409341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:42.409341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:42.414325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 306/617] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:42 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:42.416296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:42.478822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3488366-4e1d-4de0-96e0-de863bdd53c1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:42.523657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:42.524324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.524605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.524875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:42.541611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:42.543102 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:42.569532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:42.570588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:42.571585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:42.571585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:42.571585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:42.575436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 312/618] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:42.577211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:42.611720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:42.700505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7c64b0-a9b0-445f-843f-f83ca6103aae None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:42.736925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b02462-6915-4ab7-a0c7-ab0d28f39b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:42.737932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b02462-6915-4ab7-a0c7-ab0d28f39b14 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.738453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b02462-6915-4ab7-a0c7-ab0d28f39b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.738944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b02462-6915-4ab7-a0c7-ab0d28f39b14 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:42.739819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b02462-6915-4ab7-a0c7-ab0d28f39b14 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:42.740411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b02462-6915-4ab7-a0c7-ab0d28f39b14 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:42.742771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b02462-6915-4ab7-a0c7-ab0d28f39b14 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:42.759163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 307/619] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:42.770512 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:42.870922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5fcd25-c1c9-420c-8a75-d7c55c38ade3 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.871270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5fcd25-c1c9-420c-8a75-d7c55c38ade3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.871483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5fcd25-c1c9-420c-8a75-d7c55c38ade3 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:42.872957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 308/620] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:42.880103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407634f-227e-4a11-8c26-2516fbd95f8d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.880402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407634f-227e-4a11-8c26-2516fbd95f8d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.880684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407634f-227e-4a11-8c26-2516fbd95f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:42.913548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3407634f-227e-4a11-8c26-2516fbd95f8d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:42.914775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3407634f-227e-4a11-8c26-2516fbd95f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:42.920468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc658433-0d44-4d86-9bc3-1b8eda9b9f09 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:42.922630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:42.987387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:42.987941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:42.988227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:42.988499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.004269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:43.005775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:43.033704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:43.034596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:43.035472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 313/621] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:43.040611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:43.060510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 309/622] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:42 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:43.165076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da987953-c46a-430d-81fc-0ec43b13dc9c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:43.203374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c527568-6ad9-4624-8bd7-6306c8b5fdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:43.204011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c527568-6ad9-4624-8bd7-6306c8b5fdf7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:43.204299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c527568-6ad9-4624-8bd7-6306c8b5fdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:43.204578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c527568-6ad9-4624-8bd7-6306c8b5fdf7 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.205407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c527568-6ad9-4624-8bd7-6306c8b5fdf7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:43.205721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c527568-6ad9-4624-8bd7-6306c8b5fdf7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:43.207281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c527568-6ad9-4624-8bd7-6306c8b5fdf7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:43.225304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 314/623] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:43.231014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3407634f-227e-4a11-8c26-2516fbd95f8d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:43.261766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:43.314297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd424080-36a3-48df-86e4-80b735bcc046 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:43.314841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd424080-36a3-48df-86e4-80b735bcc046 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:43.314841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd424080-36a3-48df-86e4-80b735bcc046 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.317435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 315/624] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:43.321875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5175a68-67e0-4915-bead-2ac0f074aba1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:43.322155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5175a68-67e0-4915-bead-2ac0f074aba1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:43.322429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5175a68-67e0-4915-bead-2ac0f074aba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.343343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f5175a68-67e0-4915-bead-2ac0f074aba1 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:43.344817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f5175a68-67e0-4915-bead-2ac0f074aba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:43.349453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3407634f-227e-4a11-8c26-2516fbd95f8d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:43.375033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:43.403199 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:43.403810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:43.404089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:43.404357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.405135 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] RBAC: Adding query filter params (role.id=57b3843b68f846d5905474ee762f7059, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=2697ebcd1ef3492799fb7d4433b9f00f) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:43.405472 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:43.407051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:43.421739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 310/625] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:43 2022] GET /identity/v3/role_assignments?user.id=2697ebcd1ef3492799fb7d4433b9f00f&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=57b3843b68f846d5905474ee762f7059 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:43.437843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:43.551681 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 316/626] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:43 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:43.566830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:43.586493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b48aced-43e9-4df0-9805-92f800c6dfb7 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:43.642988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:43.643649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:43.643936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:43.644212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] PATH_INFO: `/v3/services/network` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.644798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:43.646287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:43.655857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:43.656837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:43.657695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 311/627] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:43 2022] GET /identity/v3/services/network => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:43.662871 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5175a68-67e0-4915-bead-2ac0f074aba1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:43.780233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5175a68-67e0-4915-bead-2ac0f074aba1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:43.817627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:43.818175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:43.818546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:43.819006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.819657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] RBAC: Adding query filter params (name=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:43.819963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:43.821370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:43.826496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 317/628] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:43 2022] GET /identity/v3/services?name=network => generated 124 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:43.832204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:43.872196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:43.889756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:43.945375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180cddd8-fb3b-417a-9c0b-4e659ee19fa3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:43.983604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:43.984220 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:43.984498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:43.984767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:43.985421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] RBAC: Adding query filter params (type=network) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:43.985728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:43.986477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08ddb3-bedf-4047-954a-ac7697b19f99 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:43.987097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:43.992785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 312/629] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:43 2022] GET /identity/v3/services?type=network => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:44.001593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:44.023713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:44.032462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:44.033123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.033443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:44.046009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:44.046739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:44.047683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:44.048587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:44.048587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:44.048587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:44.048587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 19:47:44.048587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:44.048587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 318/630] 10.222.0.66 () {56 vars in 1129 bytes} [Fri Oct 28 19:47:43 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:44.054016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:44.127762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33526465-d21e-4463-bd7e-00e69a67c6f7 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:44.154965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:44.163286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d696b2-2b92-40fd-ae17-0440d3d330ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:44.163941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d696b2-2b92-40fd-ae17-0440d3d330ba None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.164228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d696b2-2b92-40fd-ae17-0440d3d330ba None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.164505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d696b2-2b92-40fd-ae17-0440d3d330ba None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.165170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d696b2-2b92-40fd-ae17-0440d3d330ba None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:44.166588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d696b2-2b92-40fd-ae17-0440d3d330ba None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:44.203854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b147096b-67b7-4489-91fe-7e8e8db5db96 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:44.239823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43553d26-5cd1-481f-8809-00d885ece026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:44.240529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43553d26-5cd1-481f-8809-00d885ece026 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.240782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43553d26-5cd1-481f-8809-00d885ece026 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.241045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43553d26-5cd1-481f-8809-00d885ece026 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.241698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43553d26-5cd1-481f-8809-00d885ece026 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:44.242002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43553d26-5cd1-481f-8809-00d885ece026 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:44.243355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43553d26-5cd1-481f-8809-00d885ece026 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:44.248090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 319/631] 10.222.0.66 () {56 vars in 1124 bytes} [Fri Oct 28 19:47:44 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:44.252886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2b1ce1-8eeb-4f71-bf72-6d5526e75bb7 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.252886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2b1ce1-8eeb-4f71-bf72-6d5526e75bb7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.252886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2b1ce1-8eeb-4f71-bf72-6d5526e75bb7 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.252886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 320/632] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:44.253736 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2b1ce1-8eeb-4f71-bf72-6d5526e75bb7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:44.280211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 313/633] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:47:44 2022] POST /identity/v3/endpoints => generated 337 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:47:44.281494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12adf591-77f7-4e59-80f5-56faf63f3be3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.281777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12adf591-77f7-4e59-80f5-56faf63f3be3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.286913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12adf591-77f7-4e59-80f5-56faf63f3be3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.306052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-12adf591-77f7-4e59-80f5-56faf63f3be3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:44.307769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-12adf591-77f7-4e59-80f5-56faf63f3be3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:44.386276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2b1ce1-8eeb-4f71-bf72-6d5526e75bb7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:44.395232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:44.404972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 314/634] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:44 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:44.423826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:44.424460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.424735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.425008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.425698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:44.425987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:44.427486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:44.434038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 321/635] 10.222.0.66 () {56 vars in 1124 bytes} [Fri Oct 28 19:47:44 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:44.442971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12adf591-77f7-4e59-80f5-56faf63f3be3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:44.568492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12adf591-77f7-4e59-80f5-56faf63f3be3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:44.604873 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:44.605760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.606285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.606697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.607500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:44.609036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:44.687061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 315/636] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:47:44 2022] POST /identity/v3/endpoints => generated 346 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:47:44.707290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:44.711154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:44.833041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a205f139-08a9-4a2b-94ac-8028b8f312d5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:44.880301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea4384b-0dec-4156-ac63-19888d094d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:44.881578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4384b-0dec-4156-ac63-19888d094d12 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.881952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4384b-0dec-4156-ac63-19888d094d12 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.882228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4384b-0dec-4156-ac63-19888d094d12 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.882815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4384b-0dec-4156-ac63-19888d094d12 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:44.884196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4384b-0dec-4156-ac63-19888d094d12 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-6ea4384b-0dec-4156-ac63-19888d094d12 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:44.893827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:44.895085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:44.896234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:44.897834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 322/637] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:44.903997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:44.961107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ac10f-8c2e-4414-bc95-1f86c68e8b11 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.961411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ac10f-8c2e-4414-bc95-1f86c68e8b11 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.961695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ac10f-8c2e-4414-bc95-1f86c68e8b11 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.963307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 323/638] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:44.964505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:44.969072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27748f-3776-401d-a9ab-5479c388a905 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:44.969401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27748f-3776-401d-a9ab-5479c388a905 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:44.969682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27748f-3776-401d-a9ab-5479c388a905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:44.990545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4e27748f-3776-401d-a9ab-5479c388a905 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:44.991790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4e27748f-3776-401d-a9ab-5479c388a905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:45.032502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef684-1c99-4d67-b72d-b9ade7316e3f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:45.065315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:45.081347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 324/639] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:44 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:45.082786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:45.082901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:45.083097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:45.083375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:45.084537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:45.084845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:45.086345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:45.091557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 316/640] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:44 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:45.096836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e27748f-3776-401d-a9ab-5479c388a905 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:45.201526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e27748f-3776-401d-a9ab-5479c388a905 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:45.236286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bb3deb-d074-486b-af16-3f518d2f5244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:45.237223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb3deb-d074-486b-af16-3f518d2f5244 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:45.237725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb3deb-d074-486b-af16-3f518d2f5244 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:45.238216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb3deb-d074-486b-af16-3f518d2f5244 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:45.239029 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bb3deb-d074-486b-af16-3f518d2f5244 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:45.239525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bb3deb-d074-486b-af16-3f518d2f5244 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:45.241741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bb3deb-d074-486b-af16-3f518d2f5244 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:45.246968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 325/641] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:45 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:45.357076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:45.465733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f8732a-288d-44e8-9344-4f835461155b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:45.486756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:45.501132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d7d426-0d83-4557-8976-5feeb01bd7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:45.502066 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d7d426-0d83-4557-8976-5feeb01bd7dd None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:45.510648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d7d426-0d83-4557-8976-5feeb01bd7dd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:45.510648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d7d426-0d83-4557-8976-5feeb01bd7dd None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:45.510648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d7d426-0d83-4557-8976-5feeb01bd7dd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:45.510648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d7d426-0d83-4557-8976-5feeb01bd7dd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:45.596687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 317/642] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:47:45 2022] POST /identity/v3/services => generated 269 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:46.286209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2499a771-6372-4f5f-bbcf-5c643961b469 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:46.287766 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2499a771-6372-4f5f-bbcf-5c643961b469 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:46.288258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2499a771-6372-4f5f-bbcf-5c643961b469 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:46.290145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 326/643] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:46.295488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa6a9b-1390-4793-9325-0f82ab4ffd32 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:46.296027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa6a9b-1390-4793-9325-0f82ab4ffd32 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:46.296513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa6a9b-1390-4793-9325-0f82ab4ffd32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:46.315347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a3fa6a9b-1390-4793-9325-0f82ab4ffd32 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:46.316826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a3fa6a9b-1390-4793-9325-0f82ab4ffd32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:46.393049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 318/644] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:46 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:46.664207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2499a771-6372-4f5f-bbcf-5c643961b469 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:46.724432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:46.770248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2499a771-6372-4f5f-bbcf-5c643961b469 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:46.805799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:46.806431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:46.806710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:46.806981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:46.817922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:46.823788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:46.831395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:46.832247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 327/645] 10.222.0.66 () {56 vars in 1120 bytes} [Fri Oct 28 19:47:46 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:46.835413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa6a9b-1390-4793-9325-0f82ab4ffd32 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:46.950465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa6a9b-1390-4793-9325-0f82ab4ffd32 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:46.951984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:46.986675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b488dcf4-4edb-4be4-87e3-c743305a5e08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:46.987301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b488dcf4-4edb-4be4-87e3-c743305a5e08 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:46.987587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b488dcf4-4edb-4be4-87e3-c743305a5e08 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:46.987865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b488dcf4-4edb-4be4-87e3-c743305a5e08 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:46.988539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b488dcf4-4edb-4be4-87e3-c743305a5e08 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:46.988847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b488dcf4-4edb-4be4-87e3-c743305a5e08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:46.990513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b488dcf4-4edb-4be4-87e3-c743305a5e08 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:46.999808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 319/646] 10.222.0.66 () {56 vars in 1115 bytes} [Fri Oct 28 19:47:46 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:47.009392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.018043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3464273-4ef4-43d2-814f-9992d598e8e4 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.018337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3464273-4ef4-43d2-814f-9992d598e8e4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.018869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3464273-4ef4-43d2-814f-9992d598e8e4 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.021662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 320/647] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:47.035502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dff033b-7a29-4820-aee4-0c3a4a6304e4 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.035877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dff033b-7a29-4820-aee4-0c3a4a6304e4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.036161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dff033b-7a29-4820-aee4-0c3a4a6304e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.056413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-9dff033b-7a29-4820-aee4-0c3a4a6304e4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:47.057673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9dff033b-7a29-4820-aee4-0c3a4a6304e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:47.123234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f728551-81be-407e-815e-30c3fbdec356 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:47.137416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 321/648] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:47 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:47.141672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:47.158420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:47.159055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.159340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.159614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.170732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:47.172184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:47.187396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:47.188355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:47.189264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:47.189264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:47.189264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:47.189264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 328/649] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:47.196180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dff033b-7a29-4820-aee4-0c3a4a6304e4 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.298862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:47.305993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dff033b-7a29-4820-aee4-0c3a4a6304e4 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:47.342670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:47.343308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.343636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.343913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.344581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:47.344879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:47.346342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:47.360151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 322/650] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:47.366501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.425546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.476352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b3fb0-83d9-4806-b503-3595f0ba7076 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:47.513405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496d1287-cfa5-4bc5-843d-402c13137053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:47.514369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d1287-cfa5-4bc5-843d-402c13137053 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.514954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d1287-cfa5-4bc5-843d-402c13137053 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.515237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496d1287-cfa5-4bc5-843d-402c13137053 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.525196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496d1287-cfa5-4bc5-843d-402c13137053 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:47.526858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496d1287-cfa5-4bc5-843d-402c13137053 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:47.535495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e58f94-8259-45f9-985d-29ffb45ad9d4 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-496d1287-cfa5-4bc5-843d-402c13137053 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:47.541397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:47.542351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:47.543304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 329/651] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:47.544518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed618c-e5d6-46c5-876c-08b04ae1dd5d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.544806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed618c-e5d6-46c5-876c-08b04ae1dd5d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.545085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed618c-e5d6-46c5-876c-08b04ae1dd5d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.550427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 330/652] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:47.550427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afed618c-e5d6-46c5-876c-08b04ae1dd5d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.577584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3895a8fa-ce10-4eb8-a694-57df51a4d6bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:47.578329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895a8fa-ce10-4eb8-a694-57df51a4d6bc None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.578610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895a8fa-ce10-4eb8-a694-57df51a4d6bc None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.578883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895a8fa-ce10-4eb8-a694-57df51a4d6bc None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.579462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3895a8fa-ce10-4eb8-a694-57df51a4d6bc None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:47.580835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3895a8fa-ce10-4eb8-a694-57df51a4d6bc None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:47.581618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-3895a8fa-ce10-4eb8-a694-57df51a4d6bc None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:47.590445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:47.591467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:47.592305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 323/653] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:47.593085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682519e8-1156-4958-b09e-41a31cd480af None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.593569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682519e8-1156-4958-b09e-41a31cd480af None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.593749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682519e8-1156-4958-b09e-41a31cd480af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.611470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-682519e8-1156-4958-b09e-41a31cd480af None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:47.612654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-682519e8-1156-4958-b09e-41a31cd480af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:47.626910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:47.671235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afed618c-e5d6-46c5-876c-08b04ae1dd5d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:47.689358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 324/654] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:47 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:47.689358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682519e8-1156-4958-b09e-41a31cd480af None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.716272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:47.716979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.717265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.717624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.718405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:47.718712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:47.720138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:47.740060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 331/655] 10.222.0.66 () {56 vars in 1149 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:47.746371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.811462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682519e8-1156-4958-b09e-41a31cd480af None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:47.844137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:47.850971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:47.851800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.852253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.852674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.853535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:47.854041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:47.855674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:47.863289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 325/656] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:47.863399 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d12b6d-a7f8-4b5f-93d9-b51f072f98ec None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:47.867480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.887452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:47.908137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429146c4-994e-4e04-8a1d-d5ce093317fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:47.908779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429146c4-994e-4e04-8a1d-d5ce093317fb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.909073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429146c4-994e-4e04-8a1d-d5ce093317fb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.909362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429146c4-994e-4e04-8a1d-d5ce093317fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.923908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429146c4-994e-4e04-8a1d-d5ce093317fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:47.925388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429146c4-994e-4e04-8a1d-d5ce093317fb None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-429146c4-994e-4e04-8a1d-d5ce093317fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:47.943698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:47.944757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:47.945714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:47.945714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:47.945714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:47.945714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 332/657] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:47.946461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185f8be3-d029-41ad-b060-f8bb3852c056 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:47.946753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185f8be3-d029-41ad-b060-f8bb3852c056 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:47.947029 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185f8be3-d029-41ad-b060-f8bb3852c056 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:47.948391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 333/658] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:47.953013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185f8be3-d029-41ad-b060-f8bb3852c056 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:47.991513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613def64-b0c9-4909-aad9-fd51b5cb993d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.024882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c9a1f2-3829-4b02-b9ad-55450636437f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.025575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c9a1f2-3829-4b02-b9ad-55450636437f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.025863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c9a1f2-3829-4b02-b9ad-55450636437f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.026138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c9a1f2-3829-4b02-b9ad-55450636437f None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.026758 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c9a1f2-3829-4b02-b9ad-55450636437f None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:48.027059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c9a1f2-3829-4b02-b9ad-55450636437f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.028430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c9a1f2-3829-4b02-b9ad-55450636437f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.033034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 326/659] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:48.034293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb80ea3-e987-401a-9272-f239a639c5ff None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.034580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb80ea3-e987-401a-9272-f239a639c5ff None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.034852 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb80ea3-e987-401a-9272-f239a639c5ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.056032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-acb80ea3-e987-401a-9272-f239a639c5ff None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:48.057252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-acb80ea3-e987-401a-9272-f239a639c5ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:48.067533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185f8be3-d029-41ad-b060-f8bb3852c056 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.110259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.110884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.111173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.111449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.112113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:48.112419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.113858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.122201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:48.128571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 334/660] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:48.129477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:48.139354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 327/661] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:48 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:48.140527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb80ea3-e987-401a-9272-f239a639c5ff None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:48.157388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:48.251192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f6a157-de1a-4d58-a7db-b8b1212261b4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.278224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb80ea3-e987-401a-9272-f239a639c5ff None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.286785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.288398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.288685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.288964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.289651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.291030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.315025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.315661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.315944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.316224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.323206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.324617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:48.336740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:48.337692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:48.338578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 328/662] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:48.346415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:48.369358 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 335/663] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:47:48 2022] POST /identity/v3/services => generated 241 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:48.435935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:48.438154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:48.484455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f68b3a-a003-42a4-aa43-999e36e4b517 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.523228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:48.530723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78995beb-c00e-4788-8996-e00230480745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.531499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78995beb-c00e-4788-8996-e00230480745 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.531896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78995beb-c00e-4788-8996-e00230480745 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.532348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78995beb-c00e-4788-8996-e00230480745 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.533150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78995beb-c00e-4788-8996-e00230480745 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:48.533603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78995beb-c00e-4788-8996-e00230480745 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.535252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78995beb-c00e-4788-8996-e00230480745 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.549565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 329/664] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:48.556250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78995beb-c00e-4788-8996-e00230480745 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:48.584927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce85d88-138c-4ac8-8dc4-c6f59fd3fd9d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.631658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.632331 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.632607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.632879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.633500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.634998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:48.644715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:48.645711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:48.646869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 336/665] 10.222.0.66 () {56 vars in 1132 bytes} [Fri Oct 28 19:47:48 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:48.651747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:48.694477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78995beb-c00e-4788-8996-e00230480745 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.727036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:48.743179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.744548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.745245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.745705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/2697ebcd1ef3492799fb7d4433b9f00f/roles/999c9c99957049adb2fa0212e9761e1d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.749299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=2697ebcd1ef3492799fb7d4433b9f00f, role_id=999c9c99957049adb2fa0212e9761e1d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.751912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.769380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a97d86a-55ee-4b26-87fe-f83c3adb2b68 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.794195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:48.806538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.807245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.807602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.808137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.808666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:48.808987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.810545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:48.814767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 330/666] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:47:48 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/2697ebcd1ef3492799fb7d4433b9f00f/roles/999c9c99957049adb2fa0212e9761e1d => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:47:48.825366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 337/667] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:48 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:48.825581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:48.952082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f298580-632d-4d3d-b9a9-57b1a789715b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:48.990891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:48.991511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:48.991798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:48.992071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:48.992703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:48.993009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:48.994437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:49.000132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 331/668] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:48 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:49.006165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:49.120590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef695001-3cae-4c91-ae37-8f7afd0d7524 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:49.156186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf845e5-9a20-42f1-be02-16706dd26252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:49.157135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf845e5-9a20-42f1-be02-16706dd26252 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:49.157626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf845e5-9a20-42f1-be02-16706dd26252 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:49.158082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf845e5-9a20-42f1-be02-16706dd26252 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:49.158959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf845e5-9a20-42f1-be02-16706dd26252 None admin] RBAC: Adding query filter params (interface=public, service_id=b946c6ff66654408969f615f594e52a6, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:49.159464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf845e5-9a20-42f1-be02-16706dd26252 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:49.161083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf845e5-9a20-42f1-be02-16706dd26252 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:49.167969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 338/669] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:47:49 2022] GET /identity/v3/endpoints?service_id=b946c6ff66654408969f615f594e52a6&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:49.174405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:49.223747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:49.279964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa5421-2111-4519-be94-69b321e716fb None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:49.315953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6946009-c597-41cd-b3a5-662e5c8d33e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:49.316565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6946009-c597-41cd-b3a5-662e5c8d33e2 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:49.316840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6946009-c597-41cd-b3a5-662e5c8d33e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:49.317108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6946009-c597-41cd-b3a5-662e5c8d33e2 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:49.317740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6946009-c597-41cd-b3a5-662e5c8d33e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:49.319345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6946009-c597-41cd-b3a5-662e5c8d33e2 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:49.330347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 332/670] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:47:49 2022] GET /identity/v3/services => generated 1258 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 19:47:50.342212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebaffe0-39d6-4e61-b953-14ee049dcb5f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:50.342708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebaffe0-39d6-4e61-b953-14ee049dcb5f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:50.342708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebaffe0-39d6-4e61-b953-14ee049dcb5f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:50.344236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 339/671] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:50.349713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4016d28-612c-4269-b288-b70dd472e194 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:50.350259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4016d28-612c-4269-b288-b70dd472e194 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:50.350259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4016d28-612c-4269-b288-b70dd472e194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:50.368858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d4016d28-612c-4269-b288-b70dd472e194 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:50.370062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d4016d28-612c-4269-b288-b70dd472e194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:50.440390 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 333/672] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:50 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:50.621345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a925c1c3-30c7-4abb-b066-918fd443588c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:50.621706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a925c1c3-30c7-4abb-b066-918fd443588c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:50.621822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a925c1c3-30c7-4abb-b066-918fd443588c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:50.623453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 340/673] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:50.629350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32e7261-b58a-48fe-a792-a4a9c228411d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:50.629656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32e7261-b58a-48fe-a792-a4a9c228411d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:50.629928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32e7261-b58a-48fe-a792-a4a9c228411d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:50.649284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a32e7261-b58a-48fe-a792-a4a9c228411d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:50.650483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a32e7261-b58a-48fe-a792-a4a9c228411d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:50.685752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:50.718104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a925c1c3-30c7-4abb-b066-918fd443588c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:50.725465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 334/674] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:50 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:50.771712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:50.829038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a925c1c3-30c7-4abb-b066-918fd443588c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:50.868324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af4493d-fbbf-4733-a62b-239d72109b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:50.868937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af4493d-fbbf-4733-a62b-239d72109b30 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:50.869221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af4493d-fbbf-4733-a62b-239d72109b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:50.869536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af4493d-fbbf-4733-a62b-239d72109b30 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:50.870197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af4493d-fbbf-4733-a62b-239d72109b30 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:50.871611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af4493d-fbbf-4733-a62b-239d72109b30 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:50.978840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 341/675] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:47:50 2022] POST /identity/v3/services => generated 251 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:51.001788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32e7261-b58a-48fe-a792-a4a9c228411d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:51.131071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32e7261-b58a-48fe-a792-a4a9c228411d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:51.174993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:51.175966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e100154-fa3b-4707-b77f-033c9f66398c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.176214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.176318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e100154-fa3b-4707-b77f-033c9f66398c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.176518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e100154-fa3b-4707-b77f-033c9f66398c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.176986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.177559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.178603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 342/676] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:47:51.179236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:51.180871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:51.183400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a027682f-b424-4803-9810-1b2ef15944a1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.183684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a027682f-b424-4803-9810-1b2ef15944a1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.183968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a027682f-b424-4803-9810-1b2ef15944a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:51.190768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:51.191774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:51.192615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 335/677] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:51 2022] GET /identity/v3/services/object-store => generated 93 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:51.199775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:51.215410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a027682f-b424-4803-9810-1b2ef15944a1 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:51.217447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a027682f-b424-4803-9810-1b2ef15944a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:51.235615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:51.317809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 343/678] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:51.329572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0985ab1-98c3-4359-9f8a-a9e1e77b9771 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:51.368012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:51.368647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.368938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.369220 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.369885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:51.370190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:51.371574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:51.376794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 336/679] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:51 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:51.383154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a027682f-b424-4803-9810-1b2ef15944a1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:51.429621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:51.496649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a027682f-b424-4803-9810-1b2ef15944a1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:51.535364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:51.536002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.536608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.537687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.539032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:51.539344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:51.540731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:51.546387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 344/680] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:51 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:51.552110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:51.600463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:51.659868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:51.668050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1969ce2-4f23-473c-8115-1bc5c76aaede None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:51.707737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6ec509-e586-488d-bf14-29990c001e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:51.708357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ec509-e586-488d-bf14-29990c001e85 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.708654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ec509-e586-488d-bf14-29990c001e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.709581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ec509-e586-488d-bf14-29990c001e85 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.710478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ec509-e586-488d-bf14-29990c001e85 None admin] RBAC: Adding query filter params (interface=public, service_id=eadab68a82b84f9a8374545d6f401a96, region_id=RegionOne) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:51.710788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ec509-e586-488d-bf14-29990c001e85 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:51.712165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ec509-e586-488d-bf14-29990c001e85 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:51.713253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33c6689-1798-44fb-8f1c-539d20f3daec None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:51.716409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:51.721992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 337/681] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:47:51 2022] GET /identity/v3/endpoints?service_id=eadab68a82b84f9a8374545d6f401a96&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:51.721992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50e8030-5b27-4d79-93af-6e5adafdd06e None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.721992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50e8030-5b27-4d79-93af-6e5adafdd06e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.721992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50e8030-5b27-4d79-93af-6e5adafdd06e None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.725461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 338/682] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:51 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:51.726321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50e8030-5b27-4d79-93af-6e5adafdd06e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:51.751462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96704743-a5ba-4591-b466-32dafcca22ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:51.752091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96704743-a5ba-4591-b466-32dafcca22ed None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.752370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96704743-a5ba-4591-b466-32dafcca22ed None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.752685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96704743-a5ba-4591-b466-32dafcca22ed None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.763952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96704743-a5ba-4591-b466-32dafcca22ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-96704743-a5ba-4591-b466-32dafcca22ed None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:47:51.768834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:51.769791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:51.771299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 345/683] 10.222.0.66 () {56 vars in 1120 bytes} [Fri Oct 28 19:47:51 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:51.774631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d6435-95fb-4e14-9426-15b441a54751 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.775267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d6435-95fb-4e14-9426-15b441a54751 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.775557 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d6435-95fb-4e14-9426-15b441a54751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.794381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fd6d6435-95fb-4e14-9426-15b441a54751 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:51.795554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fd6d6435-95fb-4e14-9426-15b441a54751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:51.840930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50e8030-5b27-4d79-93af-6e5adafdd06e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:51.873433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 346/684] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:51.874589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6d6435-95fb-4e14-9426-15b441a54751 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:51.888872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:51.889820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:51.890335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:51.890823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:51.891803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:51.893267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:51.906993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 339/685] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:47:51 2022] GET /identity/v3/services => generated 1497 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 19:47:52.011356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:52.018241 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6d6435-95fb-4e14-9426-15b441a54751 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.056176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.056817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.057099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.057410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.058093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:52.058397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.059829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.069374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 347/686] 10.222.0.66 () {56 vars in 1115 bytes} [Fri Oct 28 19:47:51 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:52.076695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.107629 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:52.179153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.187771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922033a6-4027-4e5f-b4e0-b3c01825ebd5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.231923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.233008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.233295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.233744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.248895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.251289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:52.270779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:52.271864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:52.272756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:52.272756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:52.272756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:52.272756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 340/687] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:52.278306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.298182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d52f911-d9a7-48ac-a8f0-bbfd09f779c6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.345265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.345985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.346265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.346539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.347158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.348537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:52.357981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:52.360094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 19:47:52.362261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:52.363998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:52.364144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 348/688] 10.222.0.66 () {56 vars in 1132 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:52.369039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.385592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4f9d1e-e264-4121-b632-5ac426efec6f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.393224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:52.423708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.424248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.424535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.424797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.425482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:52.425755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.427119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.445463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 341/689] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:52.446829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.489797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3556f4f9-195e-431f-a37a-e551ff3b2706 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.526372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.527267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.527713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.528131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.528900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:52.529370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.530871 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.541548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 349/690] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:52.542965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.555421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db2e38a-2668-4c62-b4d3-1ed74b1515e9 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.600949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.602025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.602407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.603057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.605605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:52.613914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.616956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:47:52.632757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:52.633827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 19:47:52.634842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:52.636012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 342/691] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:52.640676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.648269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:52.664979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b875ee-e972-42d4-8aff-d757065fe449 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.701049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.701715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.701999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.702274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.702905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:52.703284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.704575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.710508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 350/692] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:52.716045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.756618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7dc207-ef7b-4b43-9bcf-19f32f36502b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.795013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.795922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.796429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.796968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.797901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:52.798403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.800288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.826820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 343/693] 10.222.0.66 () {56 vars in 1149 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:52.828262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:52.847800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb78e1e4-0c7b-418f-b227-0d003f4bc258 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.891190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:52.897541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.898179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.898468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.899019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.899775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:52.901801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:52.947545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2201deef-b3c5-4d07-9e5f-c255639ac9d6 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:52.984879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a59262-a876-4ca2-99e4-caffbe6d2a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:52.985797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a59262-a876-4ca2-99e4-caffbe6d2a3c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:52.986271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a59262-a876-4ca2-99e4-caffbe6d2a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:52.986707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a59262-a876-4ca2-99e4-caffbe6d2a3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:52.993531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 351/694] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:47:52 2022] POST /identity/v3/endpoints => generated 347 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:53.078947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a59262-a876-4ca2-99e4-caffbe6d2a3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:53.080338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a59262-a876-4ca2-99e4-caffbe6d2a3c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:53.142801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:53.201275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 344/695] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:47:52 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 19:47:53.207797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:53.212559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:53.293836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52862aa-3c57-4f0f-ad89-352b1ee89bd8 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:53.294165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52862aa-3c57-4f0f-ad89-352b1ee89bd8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:53.294376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52862aa-3c57-4f0f-ad89-352b1ee89bd8 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:53.295982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 345/696] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:53.301749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bdbb38-54c3-41ad-b4cc-7a2208ef4e09 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:53.302040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bdbb38-54c3-41ad-b4cc-7a2208ef4e09 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:53.302324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bdbb38-54c3-41ad-b4cc-7a2208ef4e09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:53.324601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847314bd-3486-4d83-b225-d54f724edb8c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:53.325800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-42bdbb38-54c3-41ad-b4cc-7a2208ef4e09 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:53.328184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-42bdbb38-54c3-41ad-b4cc-7a2208ef4e09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:53.367205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:53.368076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:53.368517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:53.368926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:53.380545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:53.382502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:47:53.399238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:53.400855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:47:53.401879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:47:53.401879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:47:53.401879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:53.403706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 352/697] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:53.409198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:53.417273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 346/698] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:53 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:53.464055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:53.511325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e19388-d8ed-432e-b5ae-f0a4072e1b2d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:53.546397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:53.547015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:53.547298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:53.547570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:53.548213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:53.548515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:53.549911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:53.567524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 353/699] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:53.569187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bdbb38-54c3-41ad-b4cc-7a2208ef4e09 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:53.653932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:53.672787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bdbb38-54c3-41ad-b4cc-7a2208ef4e09 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:53.691355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:53.709577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:53.710432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:53.710761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:53.711091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:53.718583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:53.720075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:47:53.732261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:53.733236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:53.740014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 347/700] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:53.745461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:53.829286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713193e8-5bc5-4cfb-8519-a8cd1224510b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:53.848918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:53.864305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce130bd0-62ec-4e92-8ff9-799aec3350a3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:53.883759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:53.885061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:53.885968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:53.886467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:53.887257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:53.889525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:53.903152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:53.904269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:53.905140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:53.911172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:53.911172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:53.911172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:53.911172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:53.911517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:53.911517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:53.911517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:53.911720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 354/701] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:53 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:53.913396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:53.933349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 348/702] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:47:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:53.935395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:53.942343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:54.054544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f7f411-a8c7-480d-938f-76461975c3d9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:54.070927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb07de3-3d7a-4e8d-aae7-1566b87a26d1 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:54.098137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:54.098801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.099089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.099363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.101020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:54.101265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:54.102737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:54.108764 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 355/703] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:53 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:54.117304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:54.121162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef9b4a1-9dac-4cd1-9c95-e2fc6101477a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:54.121758 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef9b4a1-9dac-4cd1-9c95-e2fc6101477a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.122035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef9b4a1-9dac-4cd1-9c95-e2fc6101477a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.122306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef9b4a1-9dac-4cd1-9c95-e2fc6101477a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.123078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef9b4a1-9dac-4cd1-9c95-e2fc6101477a None admin] RBAC: Adding query filter params (role.id=999c9c99957049adb2fa0212e9761e1d, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=2697ebcd1ef3492799fb7d4433b9f00f) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:54.124212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef9b4a1-9dac-4cd1-9c95-e2fc6101477a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:54.124937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef9b4a1-9dac-4cd1-9c95-e2fc6101477a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:54.141966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 349/704] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:47:53 2022] GET /identity/v3/role_assignments?user.id=2697ebcd1ef3492799fb7d4433b9f00f&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=999c9c99957049adb2fa0212e9761e1d => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:54.179463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:54.231826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ef91f2-801a-409e-a323-fb789ceda0a8 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.232123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ef91f2-801a-409e-a323-fb789ceda0a8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.232381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ef91f2-801a-409e-a323-fb789ceda0a8 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.234039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 350/705] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:47:54.239803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673d59c1-d8a0-4370-85a7-5c988b441f1a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.240109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673d59c1-d8a0-4370-85a7-5c988b441f1a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.240440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673d59c1-d8a0-4370-85a7-5c988b441f1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.246616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0d83b-529d-4649-9edb-67693764a9ee None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:54.260087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-673d59c1-d8a0-4370-85a7-5c988b441f1a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:54.261310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-673d59c1-d8a0-4370-85a7-5c988b441f1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:54.294768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:54.296461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.296941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.297674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.298359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:54.298671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:54.300138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:54.306139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 356/706] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:54 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:54.312199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:54.337890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:54.346063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 351/707] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:54.428941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c753ba-1a6d-4bf4-bda8-4275239af616 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:54.432126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:54.469647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:54.470258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.470537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.470809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.471502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] RBAC: Adding query filter params (interface=public, service_id=0fed2f6195884044a4cae45562996990, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:54.471805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:54.473163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:54.479270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 357/708] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:47:54 2022] GET /identity/v3/endpoints?service_id=0fed2f6195884044a4cae45562996990&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:54.484844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673d59c1-d8a0-4370-85a7-5c988b441f1a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:54.590355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673d59c1-d8a0-4370-85a7-5c988b441f1a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:54.625261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:54.625905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.626183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.626456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.627054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:54.628422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:54.629368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:54.636881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 352/709] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:47:54 2022] GET /identity/v3/services => generated 1497 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 19:47:54.746163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd876d45-5d61-4879-b9a8-45d7ef4aab73 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:54.785284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:54.786217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.786672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.787667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.788427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:54.790011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:54.800386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:54.813771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:54.814827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 358/710] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:54 2022] GET /identity/v3/services/object-store => generated 93 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:54.815803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:54.884212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:54.926338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba4280d-a905-4c81-8a2e-339d63f4f17a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:54.962997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:54.963616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:54.963896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:54.964166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:54.964791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:54.965092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:54.966495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:54.971883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 353/711] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:54 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:54.978053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:55.012943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:55.089660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fed51f4-5c44-40a4-a11d-10711fdb8078 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:55.129707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96e515-d24b-4166-9f16-7978babae020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:55.130652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96e515-d24b-4166-9f16-7978babae020 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:55.132218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96e515-d24b-4166-9f16-7978babae020 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:55.132693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96e515-d24b-4166-9f16-7978babae020 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:55.133536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96e515-d24b-4166-9f16-7978babae020 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:55.134022 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96e515-d24b-4166-9f16-7978babae020 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:55.135584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96e515-d24b-4166-9f16-7978babae020 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:55.141369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 359/712] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:54 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:55.153359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:55.254148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893a206a-b525-4bd3-a9d4-cd1196fd96b3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:55.281653 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:55.292912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802429fd-2350-4fe2-920f-80b457f9272b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:55.293665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802429fd-2350-4fe2-920f-80b457f9272b None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:55.293992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802429fd-2350-4fe2-920f-80b457f9272b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:55.294350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802429fd-2350-4fe2-920f-80b457f9272b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:55.295027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802429fd-2350-4fe2-920f-80b457f9272b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:55.296389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802429fd-2350-4fe2-920f-80b457f9272b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:55.371226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 354/713] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:47:55 2022] POST /identity/v3/endpoints => generated 349 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:55.377724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c9471-4f3b-476e-92ff-ef63a19071d6 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:55.378360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c9471-4f3b-476e-92ff-ef63a19071d6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:55.378847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c9471-4f3b-476e-92ff-ef63a19071d6 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:55.382210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 360/714] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:55.388459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018772c-f42b-4dad-8f20-4aa52a88b18d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:55.388808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018772c-f42b-4dad-8f20-4aa52a88b18d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:55.389087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018772c-f42b-4dad-8f20-4aa52a88b18d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:55.410811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5018772c-f42b-4dad-8f20-4aa52a88b18d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:55.412257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5018772c-f42b-4dad-8f20-4aa52a88b18d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:55.497691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 355/715] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:55.780264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394c9471-4f3b-476e-92ff-ef63a19071d6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:55.885366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:55.892201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394c9471-4f3b-476e-92ff-ef63a19071d6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:55.930230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:55.930877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:55.931202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:55.931447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:55.932031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:55.933436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:55.942938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:55.944068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:47:55.944945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:55.946594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 361/716] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:47:55 2022] GET /identity/v3/services/compute => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:47:55.951015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018772c-f42b-4dad-8f20-4aa52a88b18d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:56.065244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018772c-f42b-4dad-8f20-4aa52a88b18d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:56.073594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:56.101065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c216dd-efcc-41b7-b533-fc766899cb3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:56.101756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c216dd-efcc-41b7-b533-fc766899cb3e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:56.102036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c216dd-efcc-41b7-b533-fc766899cb3e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:56.102306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c216dd-efcc-41b7-b533-fc766899cb3e None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:56.102935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c216dd-efcc-41b7-b533-fc766899cb3e None admin] RBAC: Adding query filter params (name=compute) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:56.103233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c216dd-efcc-41b7-b533-fc766899cb3e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:56.104614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c216dd-efcc-41b7-b533-fc766899cb3e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:56.109837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 356/717] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:55 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:56.117219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:56.239565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3c876e-f045-4de9-90e3-eda3be5759c0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:56.274266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357d55fa-d85a-4e38-867c-9ebef88d8528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:56.274882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357d55fa-d85a-4e38-867c-9ebef88d8528 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:56.275166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357d55fa-d85a-4e38-867c-9ebef88d8528 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:56.275443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357d55fa-d85a-4e38-867c-9ebef88d8528 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:56.276073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357d55fa-d85a-4e38-867c-9ebef88d8528 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:56.276373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357d55fa-d85a-4e38-867c-9ebef88d8528 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:56.277851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357d55fa-d85a-4e38-867c-9ebef88d8528 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:56.283374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 362/718] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:47:56 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:56.570092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9327cf-8748-4682-8691-6547898dc2b9 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:56.570611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9327cf-8748-4682-8691-6547898dc2b9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:56.570611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9327cf-8748-4682-8691-6547898dc2b9 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:56.572197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 357/719] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:56.577471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdac719-3a89-4fe5-9b1b-e37ad64ead64 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:56.577763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdac719-3a89-4fe5-9b1b-e37ad64ead64 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:56.578041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdac719-3a89-4fe5-9b1b-e37ad64ead64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:56.597070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1bdac719-3a89-4fe5-9b1b-e37ad64ead64 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:56.598309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1bdac719-3a89-4fe5-9b1b-e37ad64ead64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:56.617433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:56.669714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 363/720] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:56.909956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b19ed-309f-44bd-9f01-e5f316ba355d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:56.910142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b19ed-309f-44bd-9f01-e5f316ba355d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:56.910424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b19ed-309f-44bd-9f01-e5f316ba355d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:56.912037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 358/721] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:56.917630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a4682-1e06-4d73-af4e-b77e67a6e20a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:56.918060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a4682-1e06-4d73-af4e-b77e67a6e20a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:56.918221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a4682-1e06-4d73-af4e-b77e67a6e20a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:56.936916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f13a4682-1e06-4d73-af4e-b77e67a6e20a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:56.938123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f13a4682-1e06-4d73-af4e-b77e67a6e20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:56.939947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b19ed-309f-44bd-9f01-e5f316ba355d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:57.040735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 364/722] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:57.057743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b19ed-309f-44bd-9f01-e5f316ba355d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:57.094396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:57.095086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.095407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.095678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] PATH_INFO: `/v3/services/image` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.096256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:57.097699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:57.107576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:57.108566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:57.109438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 359/723] 10.222.0.66 () {56 vars in 1105 bytes} [Fri Oct 28 19:47:56 2022] GET /identity/v3/services/image => generated 86 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:57.115142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13a4682-1e06-4d73-af4e-b77e67a6e20a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:57.204247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:57.221729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13a4682-1e06-4d73-af4e-b77e67a6e20a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:57.269352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:57.269352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.269855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.270007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.273342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] RBAC: Adding query filter params (name=image) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:57.273342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:57.273342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:57.278265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 365/724] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:57 2022] GET /identity/v3/services?name=image => generated 122 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:57.283825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:57.320742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:57.328660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:57.404665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd706d-ae0c-46d4-98bc-27886d92774a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:57.440518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8af283-e041-4826-a900-124752c4deef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:57.441135 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8af283-e041-4826-a900-124752c4deef None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.441437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8af283-e041-4826-a900-124752c4deef None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.441643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad85c2f-70c2-419e-8f04-4fd01de82e2e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:57.441749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8af283-e041-4826-a900-124752c4deef None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.442324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8af283-e041-4826-a900-124752c4deef None admin] RBAC: Adding query filter params (type=image) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:57.442624 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8af283-e041-4826-a900-124752c4deef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:57.444448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8af283-e041-4826-a900-124752c4deef None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:57.457577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 360/725] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:47:57 2022] GET /identity/v3/services?type=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:57.482250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b8ff9d-28e4-4c8e-a542-1a1dfb7d3f58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:57.483283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b8ff9d-28e4-4c8e-a542-1a1dfb7d3f58 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.483806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b8ff9d-28e4-4c8e-a542-1a1dfb7d3f58 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.484285 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b8ff9d-28e4-4c8e-a542-1a1dfb7d3f58 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.485101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b8ff9d-28e4-4c8e-a542-1a1dfb7d3f58 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:57.486757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b8ff9d-28e4-4c8e-a542-1a1dfb7d3f58 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-80b8ff9d-28e4-4c8e-a542-1a1dfb7d3f58 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:57.497391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:57.498452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 19:47:57.499317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:47:57.502396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 366/726] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:57 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:57.509729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8af283-e041-4826-a900-124752c4deef None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:57.626342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:57.633414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209c61d-2af0-49ed-9e58-ec1184c8187e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.633996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209c61d-2af0-49ed-9e58-ec1184c8187e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.634507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209c61d-2af0-49ed-9e58-ec1184c8187e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.635872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8af283-e041-4826-a900-124752c4deef None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:57.636592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 367/727] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:57.643100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf17a2c-af10-466a-8d4c-f96a71517b50 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.643382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf17a2c-af10-466a-8d4c-f96a71517b50 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.646881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf17a2c-af10-466a-8d4c-f96a71517b50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.670944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6bf17a2c-af10-466a-8d4c-f96a71517b50 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:57.672125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6bf17a2c-af10-466a-8d4c-f96a71517b50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:57.681853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:57.687628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:57.688530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.688824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.689100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.689827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:57.690133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:57.691687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:57.698727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 361/728] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:57 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:57.704050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:57.762989 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 368/729] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:47:57.823896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b2131e-fee1-4f73-a68c-e8013bd42543 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:57.858986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:57.859959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:57.860437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:57.860902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:57.861775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:57.862272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:57.863835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:57.869874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 362/730] 10.222.0.66 () {56 vars in 1114 bytes} [Fri Oct 28 19:47:57 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:57.878403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf17a2c-af10-466a-8d4c-f96a71517b50 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:57.983828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf17a2c-af10-466a-8d4c-f96a71517b50 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:58.020707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:58.021247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.021605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.022138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.022813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:58.024186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:58.052453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:58.083245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:58.109315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:58.116502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 369/731] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:47:57 2022] POST /identity/v3/endpoints => generated 346 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:58.167509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9317d631-0a79-4b84-9144-0a2f1a4fd6e9 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:58.214464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:58.215118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.215398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.215670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.216259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:58.217691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:47:58.228354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:47:58.229478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:47:58.230992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 363/732] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:47:58 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:47:58.238479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:58.356318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bd3c8e-240f-4484-8c34-e96e42c96855 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:58.402016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998766af-b536-4bc3-827a-ee4fc2803656 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:58.402659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998766af-b536-4bc3-827a-ee4fc2803656 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.402944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998766af-b536-4bc3-827a-ee4fc2803656 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.403287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998766af-b536-4bc3-827a-ee4fc2803656 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.404128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998766af-b536-4bc3-827a-ee4fc2803656 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:58.404465 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998766af-b536-4bc3-827a-ee4fc2803656 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:58.405979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998766af-b536-4bc3-827a-ee4fc2803656 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:58.412589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 370/733] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:58 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:58.418440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:58.518266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:58.535427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1347920e-0247-427d-a0e2-d1b6d15d732c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:58.571797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:58.572747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.572854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6bee30-af3c-495c-aefa-156f1ef82044 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.573063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6bee30-af3c-495c-aefa-156f1ef82044 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.573370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6bee30-af3c-495c-aefa-156f1ef82044 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.573763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.574221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.574989 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 371/734] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:58.575232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:58.575716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:58.577292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:58.579942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670cf685-45ec-4635-bfab-c713fd804ae6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.580217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670cf685-45ec-4635-bfab-c713fd804ae6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.580504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670cf685-45ec-4635-bfab-c713fd804ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.584370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 364/735] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:47:58 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:47:58.590215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:58.608085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-670cf685-45ec-4635-bfab-c713fd804ae6 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:58.609916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-670cf685-45ec-4635-bfab-c713fd804ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:58.683773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:58.702332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 372/736] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:58 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:47:58.713528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd3a5e0-de28-49d1-904d-808687275016 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:58.750171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:58.751082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.751537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.752010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.752900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] RBAC: Adding query filter params (interface=admin, service_id=eadab68a82b84f9a8374545d6f401a96, region_id=RegionOne) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:47:58.753398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:58.754938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:58.760694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 365/737] 10.222.0.66 () {56 vars in 1249 bytes} [Fri Oct 28 19:47:58 2022] GET /identity/v3/endpoints?service_id=eadab68a82b84f9a8374545d6f401a96&interface=admin®ion_id=RegionOne => generated 193 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:47:58.766519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670cf685-45ec-4635-bfab-c713fd804ae6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:58.869652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670cf685-45ec-4635-bfab-c713fd804ae6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:58.904738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdb2346-df5c-4fbb-8ea3-45bc2715d30d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:58.905670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdb2346-df5c-4fbb-8ea3-45bc2715d30d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:58.906126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdb2346-df5c-4fbb-8ea3-45bc2715d30d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:58.906570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdb2346-df5c-4fbb-8ea3-45bc2715d30d None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:58.907363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cdb2346-df5c-4fbb-8ea3-45bc2715d30d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:58.908889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cdb2346-df5c-4fbb-8ea3-45bc2715d30d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:58.915448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 373/738] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:47:58 2022] GET /identity/v3/services => generated 1497 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 19:47:58.988594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:47:59.007450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:59.118083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6becd202-0928-4d98-a8b4-48f7705e699f None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:47:59.157766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdf4cae-5deb-41fe-bcda-ecca2c0a5802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:47:59.158654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf4cae-5deb-41fe-bcda-ecca2c0a5802 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:59.159110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf4cae-5deb-41fe-bcda-ecca2c0a5802 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:59.159554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf4cae-5deb-41fe-bcda-ecca2c0a5802 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:59.160387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdf4cae-5deb-41fe-bcda-ecca2c0a5802 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:47:59.162028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdf4cae-5deb-41fe-bcda-ecca2c0a5802 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:47:59.258864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 366/739] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:47:58 2022] POST /identity/v3/services => generated 242 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:47:59.695374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dff4e7-24ac-42f2-b13e-5c14bae037b4 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:59.695920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dff4e7-24ac-42f2-b13e-5c14bae037b4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:59.695920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dff4e7-24ac-42f2-b13e-5c14bae037b4 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:59.697528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 374/740] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:47:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:47:59.703110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441d78f3-87b3-489c-8fd6-c3f279255646 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:47:59.703664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441d78f3-87b3-489c-8fd6-c3f279255646 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:47:59.704156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441d78f3-87b3-489c-8fd6-c3f279255646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:47:59.722621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-441d78f3-87b3-489c-8fd6-c3f279255646 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:47:59.724580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-441d78f3-87b3-489c-8fd6-c3f279255646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:47:59.800672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:47:59.808241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 367/741] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:47:59 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:00.089673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dff4e7-24ac-42f2-b13e-5c14bae037b4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:00.150599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:00.204497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dff4e7-24ac-42f2-b13e-5c14bae037b4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:00.243567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54508105-e15a-4850-aa67-f8b2fb6c6ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:00.244566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54508105-e15a-4850-aa67-f8b2fb6c6ab4 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:00.245052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54508105-e15a-4850-aa67-f8b2fb6c6ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:00.245586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54508105-e15a-4850-aa67-f8b2fb6c6ab4 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:00.246490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54508105-e15a-4850-aa67-f8b2fb6c6ab4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:00.248089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54508105-e15a-4850-aa67-f8b2fb6c6ab4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:00.316314 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 375/742] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:48:00 2022] POST /identity/v3/services => generated 242 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:00.460360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4953abc-8c15-4305-b935-d5562bce8b91 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:00.461069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4953abc-8c15-4305-b935-d5562bce8b91 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:00.461639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4953abc-8c15-4305-b935-d5562bce8b91 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:00.463546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 368/743] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:00.469899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21149800-fed3-4995-b5b3-0c09d6353a27 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:00.470511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21149800-fed3-4995-b5b3-0c09d6353a27 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:00.471016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21149800-fed3-4995-b5b3-0c09d6353a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:00.491719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-21149800-fed3-4995-b5b3-0c09d6353a27 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:00.493265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-21149800-fed3-4995-b5b3-0c09d6353a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:00.579050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 376/744] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:00 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:00.853259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4953abc-8c15-4305-b935-d5562bce8b91 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:00.962669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4953abc-8c15-4305-b935-d5562bce8b91 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:00.998514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:00.999491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.000098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.000580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.001485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:01.003162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:01.006674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING keystone.server.flask.common [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 28 19:48:01.085766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.sql.core [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 19:48:01.085766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 19:48:01.085766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [parameters: {'id': '0dd2b634893d415d889b2010d008483f', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 28 19:48:01.085766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81614) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-157bd711-29d8-4298-8b06-ea184f5054d0 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 0dd2b634893d415d889b2010d008483f.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 0dd2b634893d415d889b2010d008483f. Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:48:01.094334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 28 19:48:01.095175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application yield resource Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise self.value Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application session.commit() Oct 28 19:48:01.096136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.flush() Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application compat.raise_( Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 19:48:01.097035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 19:48:01.097999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:48:01.098804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [parameters: {'id': '0dd2b634893d415d889b2010d008483f', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:01.100269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:01.101081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 0dd2b634893d415d889b2010d008483f. Oct 28 19:48:01.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.107202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 369/745] 10.222.0.66 () {60 vars in 1140 bytes} [Fri Oct 28 19:48:00 2022] POST /identity/v3/projects => generated 262 bytes in 254 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:01.113381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21149800-fed3-4995-b5b3-0c09d6353a27 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:01.166422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:01.227095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21149800-fed3-4995-b5b3-0c09d6353a27 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:01.256451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a02bf-c7ea-45b5-af8a-ee782fe26a5e None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.257061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a02bf-c7ea-45b5-af8a-ee782fe26a5e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.257640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a02bf-c7ea-45b5-af8a-ee782fe26a5e None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.262158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 370/746] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:01 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:48:01.267027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:01.267939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd2018-ccc2-4af4-a623-84ff228b2c94 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.268152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.268264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd2018-ccc2-4af4-a623-84ff228b2c94 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.268500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd2018-ccc2-4af4-a623-84ff228b2c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.268922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.269421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.282836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:01.284949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:01.289256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b0bd2018-ccc2-4af4-a623-84ff228b2c94 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:01.290813 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b0bd2018-ccc2-4af4-a623-84ff228b2c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:48:01.298431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:01.299400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:01.300278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 377/747] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:48:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:01.308562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:01.361872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:01.385607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 371/748] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:01 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:48:01.425688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8482161c-1318-48d7-9180-43df8d487629 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:01.429764 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:01.463064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c40420f-2454-421e-9c69-308ceb682e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:01.464101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c40420f-2454-421e-9c69-308ceb682e13 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.464626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c40420f-2454-421e-9c69-308ceb682e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.465104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c40420f-2454-421e-9c69-308ceb682e13 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.465993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c40420f-2454-421e-9c69-308ceb682e13 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:01.466518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c40420f-2454-421e-9c69-308ceb682e13 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:01.468135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c40420f-2454-421e-9c69-308ceb682e13 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:01.489110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 378/749] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:48:01 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:01.606917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f22543-acc4-480d-a088-3fc61bd1b4b9 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.607245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f22543-acc4-480d-a088-3fc61bd1b4b9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.607468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f22543-acc4-480d-a088-3fc61bd1b4b9 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.609087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 372/750] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:01.614755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ef894a-f452-4411-a42d-4ccdb9ad0188 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.615122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ef894a-f452-4411-a42d-4ccdb9ad0188 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.615376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ef894a-f452-4411-a42d-4ccdb9ad0188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.635923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d1ef894a-f452-4411-a42d-4ccdb9ad0188 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:01.637144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d1ef894a-f452-4411-a42d-4ccdb9ad0188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:01.667086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f22543-acc4-480d-a088-3fc61bd1b4b9 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:01.719536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 379/751] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:01 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:01.779153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f22543-acc4-480d-a088-3fc61bd1b4b9 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:01.817953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:01.818897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.819291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.819574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.820163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:01.821803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:01.831122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:01.832210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:01.833049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 373/752] 10.222.0.66 () {56 vars in 1126 bytes} [Fri Oct 28 19:48:01 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:01.839253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ef894a-f452-4411-a42d-4ccdb9ad0188 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:01.939238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:01.948715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ef894a-f452-4411-a42d-4ccdb9ad0188 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:01.984455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:01.985074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:01.985385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:01.985665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:01.986307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:01.986614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:01.987988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:01.994601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 380/753] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:48:01 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:02.002738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:02.019469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:02.063764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:02.122569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e85e45-c484-48ff-b226-c9a2ee70f11d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:02.152181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00038a2-3d67-4728-b1b7-dec94f668f41 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:02.172666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:02.173692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.174714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.175205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.176321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:02.177091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:02.179955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:02.192408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:02.192408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.192408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.192408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.192782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:02.193018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 374/754] 10.222.0.66 () {56 vars in 1122 bytes} [Fri Oct 28 19:48:02 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:02.193460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:02.194814 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:02.203068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:02.204093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:48:02.204964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:02.209667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 381/755] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:48:02 2022] GET /identity/v3/services/compute => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:02.210927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:02.234311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:02.322064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfba606-d08c-4d34-99b4-fb61e7ecc076 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:02.345671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e8dd6-c8af-4de7-9b5b-d87d92902e71 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:02.361911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:02.362681 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.363032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.363433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.364129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:02.364503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:02.366258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:02.366765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:02.377061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 382/756] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:48:02 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:02.378132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:02.393693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:02.394579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.395312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.396330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.399003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:02.400532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:02.487759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:02.497844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 375/757] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:48:02 2022] POST /identity/v3/endpoints => generated 325 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:02.510595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbc3ec9-dbc5-4db8-a875-aa22ddcfcda4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:02.560921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8661b888-ad6b-4956-b174-888b8d69f8d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:02.561901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8661b888-ad6b-4956-b174-888b8d69f8d5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.562196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8661b888-ad6b-4956-b174-888b8d69f8d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.562474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8661b888-ad6b-4956-b174-888b8d69f8d5 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.563114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8661b888-ad6b-4956-b174-888b8d69f8d5 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:02.563422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8661b888-ad6b-4956-b174-888b8d69f8d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:02.564928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8661b888-ad6b-4956-b174-888b8d69f8d5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:02.572421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 383/758] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:48:02 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:02.587840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:02.618311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a352a7-7667-456a-9e29-5b99663b2214 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.618657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a352a7-7667-456a-9e29-5b99663b2214 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.618875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a352a7-7667-456a-9e29-5b99663b2214 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.620516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 384/759] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:02.626104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1793f5-f6da-40e6-b91d-c969ec2e9a98 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.626442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1793f5-f6da-40e6-b91d-c969ec2e9a98 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.626724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1793f5-f6da-40e6-b91d-c969ec2e9a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.648321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-eb1793f5-f6da-40e6-b91d-c969ec2e9a98 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:02.649653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-eb1793f5-f6da-40e6-b91d-c969ec2e9a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:02.707868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e9259a-c549-4513-a422-5c628381b364 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:02.735792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 385/760] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:02 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:02.748495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:02.749515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.750031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.750506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.751410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] RBAC: Adding query filter params (interface=public, service_id=24d9da06a7ef42d2bcdb107662c6b971, region_id=RegionOne) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:02.751938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:02.753568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:02.760680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 376/761] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:48:02 2022] GET /identity/v3/endpoints?service_id=24d9da06a7ef42d2bcdb107662c6b971&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:02.766717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1793f5-f6da-40e6-b91d-c969ec2e9a98 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:02.846280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:02.873707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1793f5-f6da-40e6-b91d-c969ec2e9a98 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:02.911994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:02.913191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:02.913513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:02.913792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:02.914409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:02.915779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:02.922730 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 386/762] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:48:02 2022] GET /identity/v3/services => generated 1957 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 19:48:03.024908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:03.076677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:03.138858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c1b7c3-b3a2-48af-9115-3d3900661630 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:03.173716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:03.174708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:03.175205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:03.175689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] PATH_INFO: `/v3/services/image` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:03.176512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:03.178169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:03.187672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:03.188657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:48:03.189897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:03.192415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 377/763] 10.222.0.66 () {56 vars in 1105 bytes} [Fri Oct 28 19:48:03 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:03.201342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:03.301919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deebcbd-1cdb-4c86-b585-19fdae1e9307 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:03.338005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:03.338626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:03.338907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:03.339177 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:03.339806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] RBAC: Adding query filter params (name=image) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:03.340107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:03.341497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:03.347118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 387/764] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:03 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:03.354232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:03.462896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98d6fe-c35b-4d43-a77a-cd02e03a2bfe None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:03.464752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:03.503374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:03.503994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:03.504274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:03.504548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:03.505177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] RBAC: Adding query filter params (type=image) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:03.505509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:03.507461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:03.512829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 378/765] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:03 2022] GET /identity/v3/services?type=image => generated 350 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:03.518542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:03.521630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:03.626945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad935ec4-8359-4b10-9fac-a2f1eb144eec None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:03.666215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63914da-1fef-4a3a-be59-2de568eddd59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:03.666829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63914da-1fef-4a3a-be59-2de568eddd59 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:03.669343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63914da-1fef-4a3a-be59-2de568eddd59 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:03.669343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63914da-1fef-4a3a-be59-2de568eddd59 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:03.669343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63914da-1fef-4a3a-be59-2de568eddd59 None admin] RBAC: Adding query filter params (interface=public, service_id=53e7bac432fc4e588259cd5a9748f519, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:03.669343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63914da-1fef-4a3a-be59-2de568eddd59 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:03.669848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63914da-1fef-4a3a-be59-2de568eddd59 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:03.675632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 388/766] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:48:03 2022] GET /identity/v3/endpoints?service_id=53e7bac432fc4e588259cd5a9748f519&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:03.682677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:03.797078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b95857-df5a-48f4-b292-1e3162b68463 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:03.832492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516e0ae9-731c-4758-93d8-3c7a7cd1f49b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:03.834393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e0ae9-731c-4758-93d8-3c7a7cd1f49b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:03.834393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e0ae9-731c-4758-93d8-3c7a7cd1f49b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:03.834393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e0ae9-731c-4758-93d8-3c7a7cd1f49b None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:03.834987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516e0ae9-731c-4758-93d8-3c7a7cd1f49b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:03.836369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516e0ae9-731c-4758-93d8-3c7a7cd1f49b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:03.843418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 379/767] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:48:03 2022] GET /identity/v3/services => generated 1957 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 19:48:03.868331 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ff2e1-3e7d-411b-8ec6-afa4c7dc5ddb None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:03.868781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ff2e1-3e7d-411b-8ec6-afa4c7dc5ddb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:03.869067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ff2e1-3e7d-411b-8ec6-afa4c7dc5ddb None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:03.870740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 389/768] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:03.877080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556e203-5e42-4eae-8d07-459f43cba29b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:03.877268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556e203-5e42-4eae-8d07-459f43cba29b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:03.877593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556e203-5e42-4eae-8d07-459f43cba29b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:03.900532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-4556e203-5e42-4eae-8d07-459f43cba29b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:03.902278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4556e203-5e42-4eae-8d07-459f43cba29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:03.932234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:03.982848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 380/769] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:03 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:04.273189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75ff2e1-3e7d-411b-8ec6-afa4c7dc5ddb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:04.351474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:04.387055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75ff2e1-3e7d-411b-8ec6-afa4c7dc5ddb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:04.421497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:04.422109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:04.422393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:04.422665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:04.423304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:04.424741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:04.427307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING keystone.server.flask.common [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 28 19:48:04.462776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.sql.core [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 19:48:04.462776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 28 19:48:04.462776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [parameters: {'user_id': '875e37dfd899497d9590f47a708abf10', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 28 19:48:04.462776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81613) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result.read() Oct 28 19:48:04.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:48:04.481369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application yield resource Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise self.value Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application session.commit() Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 19:48:04.482330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.session.flush() Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application compat.raise_( Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 19:48:04.483233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:48:04.484153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result.read() Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:48:04.485004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [parameters: {'user_id': '875e37dfd899497d9590f47a708abf10', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:04.487474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 28 19:48:04.489359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 28 19:48:04.491901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 28 19:48:04.491901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 28 19:48:04.491901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 19:48:04.491901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 19:48:04.491901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 28 19:48:04.491901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:04.491901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 390/770] 10.222.0.66 () {60 vars in 1131 bytes} [Fri Oct 28 19:48:04 2022] POST /identity/v3/users => generated 160 bytes in 209 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:04.500998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556e203-5e42-4eae-8d07-459f43cba29b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:04.604903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556e203-5e42-4eae-8d07-459f43cba29b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:04.642120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abb0cfd-d34e-4a06-bf5e-43c2d5ed9209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:04.642807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abb0cfd-d34e-4a06-bf5e-43c2d5ed9209 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:04.643094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abb0cfd-d34e-4a06-bf5e-43c2d5ed9209 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:04.643365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abb0cfd-d34e-4a06-bf5e-43c2d5ed9209 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:04.651554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abb0cfd-d34e-4a06-bf5e-43c2d5ed9209 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:04.653010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abb0cfd-d34e-4a06-bf5e-43c2d5ed9209 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-1abb0cfd-d34e-4a06-bf5e-43c2d5ed9209 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:48:04.665354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:04.666370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:04.667315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 381/771] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:48:04 2022] GET /identity/v3/users/cinder => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:04.674425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:04.781080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4952ac4-1efd-4107-93f2-4e93db6fcb81 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:04.813075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:04.821732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e4fec5-d45c-4558-afb4-686c6bf3d380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:04.822354 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4fec5-d45c-4558-afb4-686c6bf3d380 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:04.825236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4fec5-d45c-4558-afb4-686c6bf3d380 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:04.825236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4fec5-d45c-4558-afb4-686c6bf3d380 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:04.825236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e4fec5-d45c-4558-afb4-686c6bf3d380 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:04.825236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e4fec5-d45c-4558-afb4-686c6bf3d380 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:04.825604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e4fec5-d45c-4558-afb4-686c6bf3d380 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:04.864597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 391/772] 10.222.0.66 () {56 vars in 1101 bytes} [Fri Oct 28 19:48:04 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:04.903917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f994f90-63df-45c5-b3fb-ec02e9897370 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:04.904279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f994f90-63df-45c5-b3fb-ec02e9897370 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:04.904475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f994f90-63df-45c5-b3fb-ec02e9897370 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:04.906198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 382/773] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:04.915992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37a662b-a1f9-4e70-a1f5-5a42966a9389 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:04.916298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37a662b-a1f9-4e70-a1f5-5a42966a9389 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:04.916550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37a662b-a1f9-4e70-a1f5-5a42966a9389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:04.937088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a37a662b-a1f9-4e70-a1f5-5a42966a9389 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:04.938653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a37a662b-a1f9-4e70-a1f5-5a42966a9389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:05.014711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 392/774] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:04 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:05.287616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f3fb5-186d-49dc-8fea-aa9184a832fc None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:05.288010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f3fb5-186d-49dc-8fea-aa9184a832fc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:05.288201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f3fb5-186d-49dc-8fea-aa9184a832fc None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:05.289640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 383/775] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:05.289939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37a662b-a1f9-4e70-a1f5-5a42966a9389 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:05.299354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63c7d71-b2de-4460-9111-4b9dfe974143 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:05.299639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63c7d71-b2de-4460-9111-4b9dfe974143 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:05.299912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63c7d71-b2de-4460-9111-4b9dfe974143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:05.319676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f63c7d71-b2de-4460-9111-4b9dfe974143 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:05.320858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f63c7d71-b2de-4460-9111-4b9dfe974143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:05.336284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:05.392504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37a662b-a1f9-4e70-a1f5-5a42966a9389 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:05.393717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 384/776] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:05 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:48:05.410082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:05.427240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:05.427938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:05.428208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:05.428467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:05.429583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:05.431011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:05.432768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 393/777] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:05.438537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63c7d71-b2de-4460-9111-4b9dfe974143 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:05.542890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63c7d71-b2de-4460-9111-4b9dfe974143 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:05.578937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:05.579551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:05.579836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:05.580112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:05.580770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:05.582246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:05.663643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:05.697349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 385/778] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:48:05 2022] POST /identity/v3/projects => generated 311 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:05.775958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c92df-4886-4137-bce0-8da30520ac69 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:05.815745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:05.816456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:05.816822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:05.817024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:05.817660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:05.819750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:05.829775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:05.830943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:05.843208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 394/779] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:48:05 2022] GET /identity/v3/services/compute => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:48:05.844054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:05.844054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:05.950356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dc5256-c79c-47d4-b4e7-559e55f51519 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:05.985830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38916472-89b2-421d-a828-7d2199dc4e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:05.986470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38916472-89b2-421d-a828-7d2199dc4e26 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:05.986755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38916472-89b2-421d-a828-7d2199dc4e26 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:05.987031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38916472-89b2-421d-a828-7d2199dc4e26 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:05.987678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38916472-89b2-421d-a828-7d2199dc4e26 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:05.987983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38916472-89b2-421d-a828-7d2199dc4e26 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:05.989450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38916472-89b2-421d-a828-7d2199dc4e26 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:06.000772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 386/780] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:48:05 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:06.002641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:06.056113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:06.104980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0053a2-72af-4de4-885b-8c6482543bd7 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:06.105276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0053a2-72af-4de4-885b-8c6482543bd7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:06.105493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0053a2-72af-4de4-885b-8c6482543bd7 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:06.106542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd61af-9706-4b49-b1a2-e51a2293abb9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:06.106976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 387/781] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:48:06.111770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70876f4-0966-4e6b-97b6-e998f8005efa None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:06.112060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70876f4-0966-4e6b-97b6-e998f8005efa None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:06.112341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70876f4-0966-4e6b-97b6-e998f8005efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:06.131338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a70876f4-0966-4e6b-97b6-e998f8005efa None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:06.132546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a70876f4-0966-4e6b-97b6-e998f8005efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:06.145414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:06.146037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:06.146367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:06.146774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:06.147413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:06.147721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:06.149094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:06.154203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 395/782] 10.222.0.66 () {56 vars in 1112 bytes} [Fri Oct 28 19:48:06 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:06.159689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:06.199181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:06.207778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 388/783] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:06 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:48:06.267198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48582a5-3f32-466c-8695-b32abe703e77 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:06.280698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:06.302365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:06.303929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:06.304219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:06.304500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:06.305200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:06.306622 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:06.395165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 396/784] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:48:06 2022] POST /identity/v3/endpoints => generated 334 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:48:06.483611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70876f4-0966-4e6b-97b6-e998f8005efa None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:06.595889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70876f4-0966-4e6b-97b6-e998f8005efa None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:06.641103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:06.641862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:06.642148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:06.642420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] PATH_INFO: `/v3/services/image` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:06.643012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:06.644415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:06.654311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:06.655318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:06.656234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 389/785] 10.222.0.66 () {56 vars in 1105 bytes} [Fri Oct 28 19:48:06 2022] GET /identity/v3/services/image => generated 86 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:06.661274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:06.766663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e946e63b-2ddb-4a9e-aab6-9984b736a10b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:06.799645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:06.800260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:06.800545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:06.800818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:06.801474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] RBAC: Adding query filter params (name=image) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:06.801771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:06.803662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:06.808941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 397/786] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:06 2022] GET /identity/v3/services?name=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:06.814626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:06.875213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:06.919773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d0b965-dfdc-472f-a261-c6ac862d1b45 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:06.953732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ad7435-3e93-4e92-a250-62d75bba24dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:06.954343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad7435-3e93-4e92-a250-62d75bba24dc None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:06.954622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad7435-3e93-4e92-a250-62d75bba24dc None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:06.954892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad7435-3e93-4e92-a250-62d75bba24dc None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:06.955547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad7435-3e93-4e92-a250-62d75bba24dc None admin] RBAC: Adding query filter params (type=image) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:06.955849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad7435-3e93-4e92-a250-62d75bba24dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:06.957222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad7435-3e93-4e92-a250-62d75bba24dc None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:06.967307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 390/787] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:06 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:06.968517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:07.003491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:07.080598 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a363789b-94f3-421d-8b98-a0e7033a47c6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:07.114204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62c07e3-30b0-4345-bf14-b0bbcf226710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:07.114816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c07e3-30b0-4345-bf14-b0bbcf226710 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:07.115131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c07e3-30b0-4345-bf14-b0bbcf226710 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:07.115405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c07e3-30b0-4345-bf14-b0bbcf226710 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:07.116070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62c07e3-30b0-4345-bf14-b0bbcf226710 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:07.117462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62c07e3-30b0-4345-bf14-b0bbcf226710 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:07.201773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 398/788] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:48:06 2022] POST /identity/v3/endpoints => generated 327 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:48:07.922141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4663c-ddc7-46dd-86e8-687593635004 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:07.922677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4663c-ddc7-46dd-86e8-687593635004 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:07.922677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4663c-ddc7-46dd-86e8-687593635004 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:07.924186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 391/789] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:07.929611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82dc3f-d989-45dc-afc3-57af9c7b4926 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:07.929994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82dc3f-d989-45dc-afc3-57af9c7b4926 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:07.930170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82dc3f-d989-45dc-afc3-57af9c7b4926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:07.949186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8e82dc3f-d989-45dc-afc3-57af9c7b4926 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:07.950406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8e82dc3f-d989-45dc-afc3-57af9c7b4926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:08.031532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 399/790] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:08.304150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4663c-ddc7-46dd-86e8-687593635004 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:08.413205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4663c-ddc7-46dd-86e8-687593635004 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:08.432875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:08.447782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5db9613-68e3-442e-8f77-0268376fc3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:08.448423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db9613-68e3-442e-8f77-0268376fc3fa None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:08.455925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db9613-68e3-442e-8f77-0268376fc3fa None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:08.455925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db9613-68e3-442e-8f77-0268376fc3fa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:08.455925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5db9613-68e3-442e-8f77-0268376fc3fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:08.455925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5db9613-68e3-442e-8f77-0268376fc3fa None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:08.455925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 392/791] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:08.459448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e82dc3f-d989-45dc-afc3-57af9c7b4926 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:08.475626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:08.574394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e82dc3f-d989-45dc-afc3-57af9c7b4926 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:08.610801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:08.611416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:08.611701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:08.611979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:08.612642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:08.614115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:08.672172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab05b5e-68ca-4c17-bf6b-fee8a21e65d5 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:08.672492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab05b5e-68ca-4c17-bf6b-fee8a21e65d5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:08.672773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab05b5e-68ca-4c17-bf6b-fee8a21e65d5 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:08.674229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 393/792] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:48:08.680319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e2e9f-928e-4318-ac81-566880b31f82 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:08.680607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e2e9f-928e-4318-ac81-566880b31f82 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:08.680888 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e2e9f-928e-4318-ac81-566880b31f82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:08.687355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 400/793] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:48:08 2022] POST /identity/v3/users => generated 285 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:08.700807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-cc8e2e9f-928e-4318-ac81-566880b31f82 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:08.702089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-cc8e2e9f-928e-4318-ac81-566880b31f82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:08.778488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 394/794] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:48:09.046737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.118950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:09.150749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b09eba7-6c66-4526-ae6c-17dae8835241 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:09.185230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:09.185894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.186180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.186455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.196710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:09.202103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:09.203015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:48:09.203857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:09.205535 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 401/795] 10.222.0.66 () {56 vars in 1120 bytes} [Fri Oct 28 19:48:09 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:09.209510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8e2e9f-928e-4318-ac81-566880b31f82 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.312825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8e2e9f-928e-4318-ac81-566880b31f82 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:09.330508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:09.347376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ecb55a-bdca-476a-b42b-ec8cd273b607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:09.347989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ecb55a-bdca-476a-b42b-ec8cd273b607 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.348272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ecb55a-bdca-476a-b42b-ec8cd273b607 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.348545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ecb55a-bdca-476a-b42b-ec8cd273b607 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.349744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ecb55a-bdca-476a-b42b-ec8cd273b607 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:09.350161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ecb55a-bdca-476a-b42b-ec8cd273b607 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:09.351718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ecb55a-bdca-476a-b42b-ec8cd273b607 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:09.359771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 395/796] 10.222.0.66 () {56 vars in 1115 bytes} [Fri Oct 28 19:48:09 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:09.365488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.394913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e27fd3-8f57-4b77-89c5-6af6d324411c None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.395883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e27fd3-8f57-4b77-89c5-6af6d324411c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.397331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e27fd3-8f57-4b77-89c5-6af6d324411c None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.399831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 396/797] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:09 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:48:09.405150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb29073a-df48-4841-abf0-0286106f9118 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.405682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb29073a-df48-4841-abf0-0286106f9118 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.406133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb29073a-df48-4841-abf0-0286106f9118 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.427109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-cb29073a-df48-4841-abf0-0286106f9118 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:09.428401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-cb29073a-df48-4841-abf0-0286106f9118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:09.473118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb2477b-5c64-4b6a-8273-647064506da8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:09.502938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 397/798] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:09.505965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:09.511635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:09.512274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.512557 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.512904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.523913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:09.525407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:48:09.540162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:09.541355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:48:09.542454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:48:09.542454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:09.542454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:09.545245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 402/799] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:09.550032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb29073a-df48-4841-abf0-0286106f9118 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.652335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb29073a-df48-4841-abf0-0286106f9118 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:09.663168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:09.686823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:09.688226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.688504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.688773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.689438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:09.689741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:09.691096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:09.702561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 398/800] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:09.708146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.777007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.808411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf48e7c-380d-4577-b023-4918595b0474 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:09.845314 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:09.846116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.846509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.846884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.854672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:09.856168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:48:09.867294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:09.868416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:09.869412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 403/801] 10.222.0.66 () {56 vars in 1093 bytes} [Fri Oct 28 19:48:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:09.874759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.886981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d80647-0d9d-46b9-8e72-3b55636d253e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:09.919201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:09.923255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:09.923872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:09.924149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:09.924420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:09.925092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:09.926522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:09.960072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.sql.core [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 28 19:48:09.960072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 19:48:09.960072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [parameters: {'id': 'cda966c81ec744d6ba51570fc7f95afd', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 28 19:48:09.960072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81614) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:48:09.967637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 28 19:48:09.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application yield resource Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application next(self.gen) Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise self.value Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application session.commit() Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 19:48:09.969692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.session.flush() Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application compat.raise_( Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 19:48:09.970791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 19:48:09.971861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application conn.query(q) Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application result.read() Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 19:48:09.972815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [parameters: {'id': 'cda966c81ec744d6ba51570fc7f95afd', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 28 19:48:09.978815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:09.981217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 399/802] 10.222.0.66 () {60 vars in 1137 bytes} [Fri Oct 28 19:48:09 2022] POST /identity/v3/domains => generated 190 bytes in 192 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:48:09.983495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:09.987917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a29b00-eeba-4480-9310-12837d59fbb4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.013827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:10.031638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.032471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.032983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.033489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.034912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:10.035636 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.038451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.064341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 404/803] 10.222.0.66 () {56 vars in 1133 bytes} [Fri Oct 28 19:48:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:10.066491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:10.085203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941fd0b3-c4a3-459c-bebb-d285a2333537 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.120274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.120911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.121192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.121540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.133342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.134818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:48:10.151406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:10.152402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:48:10.153387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:48:10.153387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:10.153387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:10.153387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 400/804] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:10.158419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:10.172938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:10.176320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba6a4ad-0937-49dc-9b51-4f82e8fda61f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.215268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.215910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.216192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.216471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.226645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.228090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:48:10.243364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:10.244507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:48:10.245628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:48:10.245628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:10.245628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:10.249143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 405/805] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:48:10.249974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:10.251189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:10.267431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde4c04e-11b7-403b-9f0f-b28720f2f1ea None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.301466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.303005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.303511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.303980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.304874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:10.305510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.306993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.320820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 401/806] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:10.356784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1456bd-0424-40ac-af82-3af868394118 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.395993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.396946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.397267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.397542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.398274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:10.398582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.399982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.417926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 406/807] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:10.419222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:10.534346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d45256-0027-4cc9-92f1-93dd7832649e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.569989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.570618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.570894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.571160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.578419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.579862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:48:10.591557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:10.592686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:10.593870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 402/808] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:48:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:10.601028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:10.618387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:10.711498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5f3c92-7b1f-4a85-a838-73f0a09a925c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.743782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba1cdbb-dba9-435b-9c61-5eb803e0243f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.744379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba1cdbb-dba9-435b-9c61-5eb803e0243f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.744657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba1cdbb-dba9-435b-9c61-5eb803e0243f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.744928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba1cdbb-dba9-435b-9c61-5eb803e0243f None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.745643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba1cdbb-dba9-435b-9c61-5eb803e0243f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:10.745944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba1cdbb-dba9-435b-9c61-5eb803e0243f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.747374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba1cdbb-dba9-435b-9c61-5eb803e0243f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.758928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 407/809] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:48:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:10.764916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:10.774532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:10.866564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c409c6-4a59-43fd-9ef8-e0a3bd198688 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:10.905223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:10.905989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.906268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.906541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/d54e8c38f0f84e4287b0445947bde916/roles/999c9c99957049adb2fa0212e9761e1d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.908807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=d54e8c38f0f84e4287b0445947bde916, role_id=999c9c99957049adb2fa0212e9761e1d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:10.910258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:10.984754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0bd3bc-3f1b-4b2c-ac44-80360bda213e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.985053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0bd3bc-3f1b-4b2c-ac44-80360bda213e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.985361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0bd3bc-3f1b-4b2c-ac44-80360bda213e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.991701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 408/810] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:48:10.993131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b14334-a00b-4dbc-928c-289e7f2802b4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:10.993458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b14334-a00b-4dbc-928c-289e7f2802b4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:10.993794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b14334-a00b-4dbc-928c-289e7f2802b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:10.998912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 403/811] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:48:10 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/d54e8c38f0f84e4287b0445947bde916/roles/999c9c99957049adb2fa0212e9761e1d => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:48:11.018873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-68b14334-a00b-4dbc-928c-289e7f2802b4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:11.021941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-68b14334-a00b-4dbc-928c-289e7f2802b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:11.101950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 409/812] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:11.376184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:11.441693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:11.477786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb67c69-aaea-437c-9166-67a24af84a74 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:11.513003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:11.513676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:11.513961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:11.514238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:11.524904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:11.530262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:11.531358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:11.532412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 404/813] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:48:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:11.538298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b14334-a00b-4dbc-928c-289e7f2802b4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:11.540534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:11.636185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b14334-a00b-4dbc-928c-289e7f2802b4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:11.670967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:11.671583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:11.671868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:11.672145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:11.672808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:11.673114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:11.674533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:11.682901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 410/814] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:48:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:11.689105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:11.789513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698aaa94-1def-4c48-ada6-4780166070a9 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:11.815373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:11.824406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:11.825039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:11.825356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:11.825725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] PATH_INFO: `/v3/users/d14a5999bd964f9ea01de28005d19b18` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:11.838787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] RBAC: Authorizing `identity:get_user(user_id=d14a5999bd964f9ea01de28005d19b18)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:11.840243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:11.841911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 405/815] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:11 2022] GET /identity/v3/users/d14a5999bd964f9ea01de28005d19b18 => generated 285 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:11.847642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:12.025903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:12.063958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc348d5e-e4e9-4185-bb1f-e515e0c88fe7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:12.096862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc35b9df-478e-4a26-a3d5-ee8d33d3a7a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:12.097518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc35b9df-478e-4a26-a3d5-ee8d33d3a7a2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:12.097803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc35b9df-478e-4a26-a3d5-ee8d33d3a7a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:12.098080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc35b9df-478e-4a26-a3d5-ee8d33d3a7a2 None admin] PATH_INFO: `/v3/projects/5379862a84b947d39b98b638ecca4ce2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:12.099702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc35b9df-478e-4a26-a3d5-ee8d33d3a7a2 None admin] RBAC: Authorizing `identity:get_project(project_id=5379862a84b947d39b98b638ecca4ce2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:12.101222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc35b9df-478e-4a26-a3d5-ee8d33d3a7a2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:12.102858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 411/816] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:11 2022] GET /identity/v3/projects/5379862a84b947d39b98b638ecca4ce2 => generated 311 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:12.108741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:12.207651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81207347-8b52-4177-9255-f0d8efedc6dc None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:12.241916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d7fa80-3767-4d82-8381-18c55a2485cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:12.242576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d7fa80-3767-4d82-8381-18c55a2485cd None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:12.242857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d7fa80-3767-4d82-8381-18c55a2485cd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:12.243130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d7fa80-3767-4d82-8381-18c55a2485cd None admin] PATH_INFO: `/v3/projects/5379862a84b947d39b98b638ecca4ce2/users/d14a5999bd964f9ea01de28005d19b18/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:12.245003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d7fa80-3767-4d82-8381-18c55a2485cd None admin] RBAC: Authorizing `identity:create_grant(project_id=5379862a84b947d39b98b638ecca4ce2, user_id=d14a5999bd964f9ea01de28005d19b18, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:12.246476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d7fa80-3767-4d82-8381-18c55a2485cd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:12.320203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 406/817] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:48:12 2022] PUT /identity/v3/projects/5379862a84b947d39b98b638ecca4ce2/users/d14a5999bd964f9ea01de28005d19b18/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:48:13.163634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ccf2-f801-4f69-84fe-1e4884ba2e13 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:13.164243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ccf2-f801-4f69-84fe-1e4884ba2e13 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:13.164243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ccf2-f801-4f69-84fe-1e4884ba2e13 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:13.165565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 412/818] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:48:13.171645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ec22e-465b-4b79-be3b-cff714546c60 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:13.172211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ec22e-465b-4b79-be3b-cff714546c60 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:13.172321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ec22e-465b-4b79-be3b-cff714546c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:13.192097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f44ec22e-465b-4b79-be3b-cff714546c60 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:13.194357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f44ec22e-465b-4b79-be3b-cff714546c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:13.234300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:13.261956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 407/819] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:13.547686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484ccf2-f801-4f69-84fe-1e4884ba2e13 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:13.647999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484ccf2-f801-4f69-84fe-1e4884ba2e13 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:13.676374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:13.681633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:13.682260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:13.682541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:13.682815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:13.692869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:13.698113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:13.699270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:13.700653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 413/820] 10.222.0.66 () {56 vars in 1120 bytes} [Fri Oct 28 19:48:13 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:13.705177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ec22e-465b-4b79-be3b-cff714546c60 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:13.807443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ec22e-465b-4b79-be3b-cff714546c60 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:13.842409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:13.843018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:13.843309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:13.843584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:13.844248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:13.844554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:13.845977 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:13.854907 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 408/821] 10.222.0.66 () {56 vars in 1115 bytes} [Fri Oct 28 19:48:13 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:13.860636 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:13.961690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd5ff9-7366-4d9c-bdf7-8fdb58cd7ebe None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:13.995886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:13.996523 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:13.996807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:13.997080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.007275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:14.008708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:48:14.024379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:14.025510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:48:14.031126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:48:14.031126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:14.031126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:14.031126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 414/822] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:14.031858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:14.067311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:14.132530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758b0ad-db86-469d-82ff-6589a5727458 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:14.165452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:14.166060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.166341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.166658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.167278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:14.167583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:14.168970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:14.180967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 409/823] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:14.186592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:14.235273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:14.289602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aab4504-bee9-4395-8545-3bbce7f2c726 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:14.323449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3463cd8-b89c-432d-91b4-f29daa7b26cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:14.324052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3463cd8-b89c-432d-91b4-f29daa7b26cc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.324332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3463cd8-b89c-432d-91b4-f29daa7b26cc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.324604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3463cd8-b89c-432d-91b4-f29daa7b26cc None admin] PATH_INFO: `/v3/users/nova` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.332858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3463cd8-b89c-432d-91b4-f29daa7b26cc None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:14.334978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3463cd8-b89c-432d-91b4-f29daa7b26cc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f3463cd8-b89c-432d-91b4-f29daa7b26cc None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:48:14.347824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:14.349074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 19:48:14.350709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:14.353480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 415/824] 10.222.0.66 () {56 vars in 1093 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:48:14.355018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:14.440896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e4dd3-97e9-478f-8806-c5ffbb63d3fc None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.441181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e4dd3-97e9-478f-8806-c5ffbb63d3fc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.441501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e4dd3-97e9-478f-8806-c5ffbb63d3fc None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.443473 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 416/825] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:48:14.448811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b3e57-8347-4e5a-8874-918555cac5b6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.449100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b3e57-8347-4e5a-8874-918555cac5b6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.449415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b3e57-8347-4e5a-8874-918555cac5b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.449994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:14.456057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2023b5e1-4fd8-44a4-97c8-975715a2614e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:14.467904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6a1b3e57-8347-4e5a-8874-918555cac5b6 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:14.469106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6a1b3e57-8347-4e5a-8874-918555cac5b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:14.490515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:14.491111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.491387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.491658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.492369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:14.492667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:14.494123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:14.511750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 410/826] 10.222.0.66 () {56 vars in 1133 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:14.517208 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:14.538104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 417/827] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:48:14.622633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b406e-723f-46ce-abdf-ec0817433291 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:14.654938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:14.655539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.655821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.656094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.680209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:14.723936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:14.726114 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:14.819413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1b3e57-8347-4e5a-8874-918555cac5b6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:14.825630 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 411/828] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 19:48:14.831836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:14.852634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:14.881192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:14.918738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1b3e57-8347-4e5a-8874-918555cac5b6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:14.923883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b43e6f-eff8-4411-8c84-1a856b7d5b28 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:14.954491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:14.955124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.955408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.955739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.959004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:14.959702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:14.959978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:14.960246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:14.966770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:14.970232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:14.971679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:14.972710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:14.973603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:14.974318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:14.974388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 418/829] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:48:14.979293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:48:14.987891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:14.988844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:48:14.995206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:48:14.995206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:48:14.995206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:14.995206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 412/830] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:14.995206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.080191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:15.087270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058de3b4-e892-4620-9880-ddcc1fcea3bb None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.095944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bcfc9d-9e5e-4d51-85ae-a2c11948bb9b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.104124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:15.121607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.122220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.122503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.122775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.123431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:15.124016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.125495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.136219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.136219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.136219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.136219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.136610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:15.136610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.136770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 419/831] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:15.136861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.139873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.253791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff72c2f-f1a9-418b-bd43-2ad8e562ea3a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.264712 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 413/832] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:15.270346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.290780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.291423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.291712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.291993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] PATH_INFO: `/v3/users/d14a5999bd964f9ea01de28005d19b18` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.293120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] RBAC: Authorizing `identity:get_user(user_id=d14a5999bd964f9ea01de28005d19b18)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.294591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.296155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 420/833] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:15 2022] GET /identity/v3/users/d14a5999bd964f9ea01de28005d19b18 => generated 285 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:15.301777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.327735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:15.370000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de507cd-2f9e-4120-ab21-45df52b0367a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.402361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eced267-0ffb-4717-b4c3-d992efb1256e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.404369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.405002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.405282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.405585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.412284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.413833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:48:15.424844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:15.425779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:15.427419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 414/834] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:48:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:48:15.431724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.435267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.435878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.436155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.436427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.460274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:15.503297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.504731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.527888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:15.532173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c04e07-eb11-4f9e-a708-e8a19d2eb8e5 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.566069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.566676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.566958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.567228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.567955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:15.568495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.569958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.581154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 415/835] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:48:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:15.587951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.602371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 421/836] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:48:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 19:48:15.608097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.673911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:15.689083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af72bad-24fb-48d4-a46f-5bc9ea980cc4 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.704191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb82877-8a5b-46f1-b487-f27132edcc4c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.723107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.723719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.724007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.724444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.725389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] RBAC: Adding query filter params (role.id=999c9c99957049adb2fa0212e9761e1d, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=d54e8c38f0f84e4287b0445947bde916) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:15.725734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.727164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.727710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:15.732846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 416/837] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:48:15 2022] GET /identity/v3/role_assignments?user.id=d54e8c38f0f84e4287b0445947bde916&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=999c9c99957049adb2fa0212e9761e1d => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:15.738833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364d8d6-ff14-4b3c-9d35-dca9653b7ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.739483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364d8d6-ff14-4b3c-9d35-dca9653b7ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.740277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364d8d6-ff14-4b3c-9d35-dca9653b7ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.740692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364d8d6-ff14-4b3c-9d35-dca9653b7ae6 None admin] PATH_INFO: `/v3/projects/5379862a84b947d39b98b638ecca4ce2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.741852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f364d8d6-ff14-4b3c-9d35-dca9653b7ae6 None admin] RBAC: Authorizing `identity:get_project(project_id=5379862a84b947d39b98b638ecca4ce2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.743302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f364d8d6-ff14-4b3c-9d35-dca9653b7ae6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.745012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 422/838] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:15 2022] GET /identity/v3/projects/5379862a84b947d39b98b638ecca4ce2 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:15.751418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:15.863704 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d0fb89-2283-4f80-94b7-15ec63d0cda1 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:15.899751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d693e571-ae71-4e64-95cc-52ed0d437e6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:15.900313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d693e571-ae71-4e64-95cc-52ed0d437e6b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:15.900591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d693e571-ae71-4e64-95cc-52ed0d437e6b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:15.900862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d693e571-ae71-4e64-95cc-52ed0d437e6b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:15.901607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d693e571-ae71-4e64-95cc-52ed0d437e6b None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=5379862a84b947d39b98b638ecca4ce2, user.id=d14a5999bd964f9ea01de28005d19b18) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:15.901914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d693e571-ae71-4e64-95cc-52ed0d437e6b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:15.903346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d693e571-ae71-4e64-95cc-52ed0d437e6b None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:15.909155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 417/839] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:48:15 2022] GET /identity/v3/role_assignments?user.id=d14a5999bd964f9ea01de28005d19b18&scope.project.id=5379862a84b947d39b98b638ecca4ce2&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:18.125961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b8e88-5860-40ab-9d2c-2ab6b65937be None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:18.126474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b8e88-5860-40ab-9d2c-2ab6b65937be None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:18.126586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b8e88-5860-40ab-9d2c-2ab6b65937be None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:18.127888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 423/840] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:18.134189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7257d48b-6c34-476c-a752-4d8439a7237f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:18.134612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7257d48b-6c34-476c-a752-4d8439a7237f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:18.134730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7257d48b-6c34-476c-a752-4d8439a7237f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:18.152958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-7257d48b-6c34-476c-a752-4d8439a7237f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:18.154438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7257d48b-6c34-476c-a752-4d8439a7237f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:18.163517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:18.219288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 418/841] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:18.486946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b8e88-5860-40ab-9d2c-2ab6b65937be None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:18.586332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b8e88-5860-40ab-9d2c-2ab6b65937be None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:18.620504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bde3c3a-28d0-4c06-8956-41daa6626a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:18.621125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bde3c3a-28d0-4c06-8956-41daa6626a4f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:18.621434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bde3c3a-28d0-4c06-8956-41daa6626a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:18.621711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bde3c3a-28d0-4c06-8956-41daa6626a4f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:18.623291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bde3c3a-28d0-4c06-8956-41daa6626a4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:18.624717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bde3c3a-28d0-4c06-8956-41daa6626a4f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:18.626308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 424/842] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:18.632855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7257d48b-6c34-476c-a752-4d8439a7237f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:18.729217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7257d48b-6c34-476c-a752-4d8439a7237f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:18.771277 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f893c637-f9f4-47e6-8188-55176905897e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:18.771860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893c637-f9f4-47e6-8188-55176905897e None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:18.772141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893c637-f9f4-47e6-8188-55176905897e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:18.772414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893c637-f9f4-47e6-8188-55176905897e None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:18.773064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f893c637-f9f4-47e6-8188-55176905897e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:18.774523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f893c637-f9f4-47e6-8188-55176905897e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:18.875271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:18.880228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 419/843] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:48:18 2022] POST /identity/v3/users => generated 286 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 19:48:21.065193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c6314-a2a8-4b58-8861-5f5111e7b726 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:21.065726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c6314-a2a8-4b58-8861-5f5111e7b726 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:21.065820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c6314-a2a8-4b58-8861-5f5111e7b726 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:21.067124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 425/844] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:21.072855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d0095c-5281-4753-b3f4-274fe8f0b7f7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:21.073265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d0095c-5281-4753-b3f4-274fe8f0b7f7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:21.073452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d0095c-5281-4753-b3f4-274fe8f0b7f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:21.091617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-11d0095c-5281-4753-b3f4-274fe8f0b7f7 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:21.092786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-11d0095c-5281-4753-b3f4-274fe8f0b7f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:21.156634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 420/845] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:21.423197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604c6314-a2a8-4b58-8861-5f5111e7b726 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:21.482511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:21.522513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604c6314-a2a8-4b58-8861-5f5111e7b726 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:21.555225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:21.555846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:21.556129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:21.556405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] PATH_INFO: `/v3/roles/335ad63ce3214c9aa7e4f4f058f97465` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:21.557802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] RBAC: Authorizing `identity:get_role(role_id=335ad63ce3214c9aa7e4f4f058f97465)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:21.559167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:21.560466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 426/846] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:21 2022] GET /identity/v3/roles/335ad63ce3214c9aa7e4f4f058f97465 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:21.566534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d0095c-5281-4753-b3f4-274fe8f0b7f7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:21.662497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d0095c-5281-4753-b3f4-274fe8f0b7f7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:21.690094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:21.695766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681a3536-814a-435e-b612-df398057ca7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:21.696392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681a3536-814a-435e-b612-df398057ca7d None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:21.704012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681a3536-814a-435e-b612-df398057ca7d None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:21.704012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681a3536-814a-435e-b612-df398057ca7d None admin] PATH_INFO: `/v3/users/6611476c03d746f683d5f23a9c605146` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:21.710239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681a3536-814a-435e-b612-df398057ca7d None admin] RBAC: Authorizing `identity:get_user(user_id=6611476c03d746f683d5f23a9c605146)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:21.711687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681a3536-814a-435e-b612-df398057ca7d None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:21.713276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 421/847] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:21 2022] GET /identity/v3/users/6611476c03d746f683d5f23a9c605146 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:21.719083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:21.817499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7f1d3-cba4-4fad-9f5f-b93dd5ed25c9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:21.844534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:21.852848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77427b30-5e65-4720-9a6f-b371ea9bfb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:21.853503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77427b30-5e65-4720-9a6f-b371ea9bfb12 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:21.853796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77427b30-5e65-4720-9a6f-b371ea9bfb12 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:21.854072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77427b30-5e65-4720-9a6f-b371ea9bfb12 None admin] PATH_INFO: `/v3/projects/5379862a84b947d39b98b638ecca4ce2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:21.855207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77427b30-5e65-4720-9a6f-b371ea9bfb12 None admin] RBAC: Authorizing `identity:get_project(project_id=5379862a84b947d39b98b638ecca4ce2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:21.856689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77427b30-5e65-4720-9a6f-b371ea9bfb12 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:21.858319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 427/848] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:21 2022] GET /identity/v3/projects/5379862a84b947d39b98b638ecca4ce2 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:21.864317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681a3536-814a-435e-b612-df398057ca7d None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:21.961299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681a3536-814a-435e-b612-df398057ca7d None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:21.993368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadfe382-2187-4a49-a8b3-c557f5c537dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:21.994009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadfe382-2187-4a49-a8b3-c557f5c537dd None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:21.994291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadfe382-2187-4a49-a8b3-c557f5c537dd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:21.994566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadfe382-2187-4a49-a8b3-c557f5c537dd None admin] PATH_INFO: `/v3/projects/5379862a84b947d39b98b638ecca4ce2/users/6611476c03d746f683d5f23a9c605146/roles/335ad63ce3214c9aa7e4f4f058f97465` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:21.996670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadfe382-2187-4a49-a8b3-c557f5c537dd None admin] RBAC: Authorizing `identity:create_grant(project_id=5379862a84b947d39b98b638ecca4ce2, user_id=6611476c03d746f683d5f23a9c605146, role_id=335ad63ce3214c9aa7e4f4f058f97465)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:21.998130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadfe382-2187-4a49-a8b3-c557f5c537dd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:22.086655 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 422/849] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:48:21 2022] PUT /identity/v3/projects/5379862a84b947d39b98b638ecca4ce2/users/6611476c03d746f683d5f23a9c605146/roles/335ad63ce3214c9aa7e4f4f058f97465 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:48:24.189426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaf02d7-35ce-4c3e-8438-a579334ee547 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:24.189988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaf02d7-35ce-4c3e-8438-a579334ee547 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:24.189988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaf02d7-35ce-4c3e-8438-a579334ee547 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:24.191310 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 428/850] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:24.199005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f927d384-1271-4593-a0c6-241910fd60f8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:24.199433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f927d384-1271-4593-a0c6-241910fd60f8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:24.199549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f927d384-1271-4593-a0c6-241910fd60f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:24.218464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f927d384-1271-4593-a0c6-241910fd60f8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:24.219632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f927d384-1271-4593-a0c6-241910fd60f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:24.260984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:24.283409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 423/851] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:24.562662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaf02d7-35ce-4c3e-8438-a579334ee547 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:24.691966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaf02d7-35ce-4c3e-8438-a579334ee547 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:24.730147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:24.731216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:24.731727 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:24.732164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] PATH_INFO: `/v3/roles/335ad63ce3214c9aa7e4f4f058f97465` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:24.733933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] RBAC: Authorizing `identity:get_role(role_id=335ad63ce3214c9aa7e4f4f058f97465)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:24.736009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:24.737718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 429/852] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:24 2022] GET /identity/v3/roles/335ad63ce3214c9aa7e4f4f058f97465 => generated 223 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:24.744064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f927d384-1271-4593-a0c6-241910fd60f8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:24.851803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f927d384-1271-4593-a0c6-241910fd60f8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:24.889050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:24.889709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:24.890019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:24.890294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] PATH_INFO: `/v3/users/6611476c03d746f683d5f23a9c605146` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:24.891419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] RBAC: Authorizing `identity:get_user(user_id=6611476c03d746f683d5f23a9c605146)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:24.892853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:24.894476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 424/853] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:24 2022] GET /identity/v3/users/6611476c03d746f683d5f23a9c605146 => generated 286 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:24.902939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:24.975547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:25.023999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6ce661-8207-4020-a5fd-9042be748d03 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:25.060853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:25.061502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:25.061787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:25.062062 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:25.128702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:25.130094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:25.187894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:25.226272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 430/854] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:48:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 19:48:25.232425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:25.277717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:25.329511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b475b1-ee06-4d0c-ad45-07e150df2879 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:25.362672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6045af7-85b6-4eec-8a81-d4dee4560f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:25.363295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6045af7-85b6-4eec-8a81-d4dee4560f28 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:25.363579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6045af7-85b6-4eec-8a81-d4dee4560f28 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:25.363851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6045af7-85b6-4eec-8a81-d4dee4560f28 None admin] PATH_INFO: `/v3/projects/5379862a84b947d39b98b638ecca4ce2` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:25.365510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6045af7-85b6-4eec-8a81-d4dee4560f28 None admin] RBAC: Authorizing `identity:get_project(project_id=5379862a84b947d39b98b638ecca4ce2)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:25.367003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6045af7-85b6-4eec-8a81-d4dee4560f28 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:25.368616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 425/855] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:25 2022] GET /identity/v3/projects/5379862a84b947d39b98b638ecca4ce2 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:25.374895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:25.478875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ee263a-ba40-4542-a03b-1e34c5bbe7b6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:25.513072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a09efc-1d64-4b71-8416-080c80c12ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:25.513723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a09efc-1d64-4b71-8416-080c80c12ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:25.514010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a09efc-1d64-4b71-8416-080c80c12ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:25.514291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a09efc-1d64-4b71-8416-080c80c12ea5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:25.515011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a09efc-1d64-4b71-8416-080c80c12ea5 None admin] RBAC: Adding query filter params (role.id=335ad63ce3214c9aa7e4f4f058f97465, scope.project.id=5379862a84b947d39b98b638ecca4ce2, user.id=6611476c03d746f683d5f23a9c605146) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:25.515326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a09efc-1d64-4b71-8416-080c80c12ea5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:25.516747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a09efc-1d64-4b71-8416-080c80c12ea5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:25.522841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 431/856] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:48:25 2022] GET /identity/v3/role_assignments?user.id=6611476c03d746f683d5f23a9c605146&scope.project.id=5379862a84b947d39b98b638ecca4ce2&role.id=335ad63ce3214c9aa7e4f4f058f97465 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:27.678065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9325d6-e4ee-4c0e-9a60-152848bcb78a None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:27.678622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9325d6-e4ee-4c0e-9a60-152848bcb78a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:27.678622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9325d6-e4ee-4c0e-9a60-152848bcb78a None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:27.679938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 426/857] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:27.685207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81ee20-e27b-4f27-84ef-420c4265d6a8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:27.685625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81ee20-e27b-4f27-84ef-420c4265d6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:27.685791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81ee20-e27b-4f27-84ef-420c4265d6a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:27.703919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3b81ee20-e27b-4f27-84ef-420c4265d6a8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:27.705085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b81ee20-e27b-4f27-84ef-420c4265d6a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:27.728108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:27.769695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 432/858] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:28.039484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9325d6-e4ee-4c0e-9a60-152848bcb78a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:28.138776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9325d6-e4ee-4c0e-9a60-152848bcb78a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:28.152913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:28.172894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43a45db-2762-4713-a434-ab1cfa59d8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:28.173544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43a45db-2762-4713-a434-ab1cfa59d8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:28.173827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43a45db-2762-4713-a434-ab1cfa59d8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:28.174101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43a45db-2762-4713-a434-ab1cfa59d8d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:28.175191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43a45db-2762-4713-a434-ab1cfa59d8d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:28.176629 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43a45db-2762-4713-a434-ab1cfa59d8d7 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:28.178261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 427/859] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:28.184014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81ee20-e27b-4f27-84ef-420c4265d6a8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:28.281726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81ee20-e27b-4f27-84ef-420c4265d6a8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:28.314392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3441f0d-342b-4691-8ce3-d953561dc242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:28.314989 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3441f0d-342b-4691-8ce3-d953561dc242 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:28.315272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3441f0d-342b-4691-8ce3-d953561dc242 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:28.315547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3441f0d-342b-4691-8ce3-d953561dc242 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:28.316190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3441f0d-342b-4691-8ce3-d953561dc242 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:28.317628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3441f0d-342b-4691-8ce3-d953561dc242 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:28.384138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 433/860] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:48:28 2022] POST /identity/v3/projects => generated 311 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:30.543431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869cadcd-df8f-4325-b2cb-9bd047ca115d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:30.543964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869cadcd-df8f-4325-b2cb-9bd047ca115d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:30.543964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869cadcd-df8f-4325-b2cb-9bd047ca115d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:30.545294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 428/861] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:30.550516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a6b36-30dd-4775-a5b9-eb5b4467b90c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:30.550928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a6b36-30dd-4775-a5b9-eb5b4467b90c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:30.551084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a6b36-30dd-4775-a5b9-eb5b4467b90c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:30.570524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3d8a6b36-30dd-4775-a5b9-eb5b4467b90c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:30.571687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3d8a6b36-30dd-4775-a5b9-eb5b4467b90c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:30.574326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:30.637426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 434/862] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:30.932824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869cadcd-df8f-4325-b2cb-9bd047ca115d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:31.032450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869cadcd-df8f-4325-b2cb-9bd047ca115d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:31.065306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b23ce30-700a-4345-96d3-c9f89599a411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:31.065951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23ce30-700a-4345-96d3-c9f89599a411 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:31.066232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23ce30-700a-4345-96d3-c9f89599a411 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:31.066506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23ce30-700a-4345-96d3-c9f89599a411 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:31.067610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b23ce30-700a-4345-96d3-c9f89599a411 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:31.069605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b23ce30-700a-4345-96d3-c9f89599a411 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:31.071153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 429/863] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:48:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:31.076919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8a6b36-30dd-4775-a5b9-eb5b4467b90c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:31.171921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8a6b36-30dd-4775-a5b9-eb5b4467b90c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:31.204491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ca038d-44d4-4a08-936b-9a56650588d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:31.205085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ca038d-44d4-4a08-936b-9a56650588d8 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:31.205387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ca038d-44d4-4a08-936b-9a56650588d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:31.205659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ca038d-44d4-4a08-936b-9a56650588d8 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:31.206296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ca038d-44d4-4a08-936b-9a56650588d8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:31.207711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ca038d-44d4-4a08-936b-9a56650588d8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:31.219037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:31.297178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 435/864] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:48:31 2022] POST /identity/v3/users => generated 286 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:33.532110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd898a0-6641-487d-8c82-b93bd4c55f6a None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:33.532631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd898a0-6641-487d-8c82-b93bd4c55f6a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:33.532745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd898a0-6641-487d-8c82-b93bd4c55f6a None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:33.534032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 430/865] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:33.539834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808003b9-dd09-4636-a21a-65045be56a44 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:33.540271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808003b9-dd09-4636-a21a-65045be56a44 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:33.540397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808003b9-dd09-4636-a21a-65045be56a44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:33.558811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-808003b9-dd09-4636-a21a-65045be56a44 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:33.559984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-808003b9-dd09-4636-a21a-65045be56a44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:33.624031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 436/866] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:33.893019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd898a0-6641-487d-8c82-b93bd4c55f6a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:33.943559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:33.992640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd898a0-6641-487d-8c82-b93bd4c55f6a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:34.025925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:34.026557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:34.026839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:34.027114 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:34.037999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:34.042839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:34.043916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:34.046465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 431/867] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:48:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:34.049872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808003b9-dd09-4636-a21a-65045be56a44 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:34.146364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808003b9-dd09-4636-a21a-65045be56a44 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:34.171676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:34.180766 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:34.181398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:34.181685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:34.181962 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:34.182623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:34.182929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:34.184300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:34.192348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 437/868] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:48:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:34.199395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:34.299883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d27bd6-24bf-4360-966d-1056a688ca56 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:34.309853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:34.332720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:34.333914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:34.334395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:34.334679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] PATH_INFO: `/v3/users/cf2b2689ad7546168e08dcb4864a16b0` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:34.347709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] RBAC: Authorizing `identity:get_user(user_id=cf2b2689ad7546168e08dcb4864a16b0)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:34.349921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:34.351627 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 432/869] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:34 2022] GET /identity/v3/users/cf2b2689ad7546168e08dcb4864a16b0 => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:34.357357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:34.452798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5b58d-7fed-4c0c-b3f4-b61e418bf207 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:34.483825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84621c37-c5d4-445a-85f9-718b4f168901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:34.484435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84621c37-c5d4-445a-85f9-718b4f168901 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:34.484714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84621c37-c5d4-445a-85f9-718b4f168901 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:34.484984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84621c37-c5d4-445a-85f9-718b4f168901 None admin] PATH_INFO: `/v3/projects/3352dfb8b28f41debbb48e69491e79b2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:34.486088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84621c37-c5d4-445a-85f9-718b4f168901 None admin] RBAC: Authorizing `identity:get_project(project_id=3352dfb8b28f41debbb48e69491e79b2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:34.487493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84621c37-c5d4-445a-85f9-718b4f168901 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:34.489066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 438/870] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:34 2022] GET /identity/v3/projects/3352dfb8b28f41debbb48e69491e79b2 => generated 311 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:34.495607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:34.592236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093c4d3-5331-47f5-9f55-914eaa251038 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:34.625804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d527e3d3-8ac3-4fdb-877e-253708477615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:34.626443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d527e3d3-8ac3-4fdb-877e-253708477615 None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:34.626721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d527e3d3-8ac3-4fdb-877e-253708477615 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:34.626998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d527e3d3-8ac3-4fdb-877e-253708477615 None admin] PATH_INFO: `/v3/projects/3352dfb8b28f41debbb48e69491e79b2/users/cf2b2689ad7546168e08dcb4864a16b0/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:34.628838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d527e3d3-8ac3-4fdb-877e-253708477615 None admin] RBAC: Authorizing `identity:create_grant(project_id=3352dfb8b28f41debbb48e69491e79b2, user_id=cf2b2689ad7546168e08dcb4864a16b0, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:34.630380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d527e3d3-8ac3-4fdb-877e-253708477615 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:34.702316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 433/871] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:48:34 2022] PUT /identity/v3/projects/3352dfb8b28f41debbb48e69491e79b2/users/cf2b2689ad7546168e08dcb4864a16b0/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 19:48:36.839887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97479caa-43a3-4ccb-b66a-281752d38b27 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:36.840453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97479caa-43a3-4ccb-b66a-281752d38b27 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:36.840453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97479caa-43a3-4ccb-b66a-281752d38b27 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:36.841825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 439/872] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:36.847203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189fb18-b7be-456d-a87c-baab9d56f489 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:36.847602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189fb18-b7be-456d-a87c-baab9d56f489 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:36.847787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189fb18-b7be-456d-a87c-baab9d56f489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:36.866919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2189fb18-b7be-456d-a87c-baab9d56f489 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:36.868092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2189fb18-b7be-456d-a87c-baab9d56f489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:36.889985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:36.932924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 434/873] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:37.200200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97479caa-43a3-4ccb-b66a-281752d38b27 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:37.257057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:37.298959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97479caa-43a3-4ccb-b66a-281752d38b27 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:37.331610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:37.332221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:37.332500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:37.332771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:37.342884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:37.347648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:37.348781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:37.350020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 440/874] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:48:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:37.354916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189fb18-b7be-456d-a87c-baab9d56f489 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:37.450082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189fb18-b7be-456d-a87c-baab9d56f489 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:37.482145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:37.482675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:37.482953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:37.483223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:37.483868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:37.484167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:37.485550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:37.493825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 435/875] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:48:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:37.499645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:37.596740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65c51a-cdab-4d91-9b0c-d2cc295d65fb None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:37.612486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:37.629802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:37.630952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:37.631237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:37.631513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] PATH_INFO: `/v3/users/cf2b2689ad7546168e08dcb4864a16b0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:37.632634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] RBAC: Authorizing `identity:get_user(user_id=cf2b2689ad7546168e08dcb4864a16b0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:37.634100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:37.635654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 441/876] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:37 2022] GET /identity/v3/users/cf2b2689ad7546168e08dcb4864a16b0 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:37.641381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:37.663682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:37.739283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e673552-34e2-408f-8f3b-087124ee1906 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:37.771822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:37.772422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:37.772704 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:37.772982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:37.837695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:37.839133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:37.858882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:37.933266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 436/877] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:48:37 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 19:48:37.939388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:38.036549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd0e771-0031-4b79-aad5-b4239f173df5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:38.070440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82964de4-3445-4989-9979-4740d57117a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:38.071057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82964de4-3445-4989-9979-4740d57117a5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:38.071338 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82964de4-3445-4989-9979-4740d57117a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:38.071609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82964de4-3445-4989-9979-4740d57117a5 None admin] PATH_INFO: `/v3/projects/3352dfb8b28f41debbb48e69491e79b2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:38.072698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82964de4-3445-4989-9979-4740d57117a5 None admin] RBAC: Authorizing `identity:get_project(project_id=3352dfb8b28f41debbb48e69491e79b2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:38.074135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82964de4-3445-4989-9979-4740d57117a5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:38.075716 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 442/878] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:37 2022] GET /identity/v3/projects/3352dfb8b28f41debbb48e69491e79b2 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:38.081773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:38.175138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34771f6-045f-4087-9b5e-ebc5cacf6d25 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:38.195147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:38.208195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7094b13-8800-49db-9e0a-7935a41e8fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:38.212555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7094b13-8800-49db-9e0a-7935a41e8fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:38.212555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7094b13-8800-49db-9e0a-7935a41e8fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:38.212555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7094b13-8800-49db-9e0a-7935a41e8fc3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:38.212555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7094b13-8800-49db-9e0a-7935a41e8fc3 None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=3352dfb8b28f41debbb48e69491e79b2, user.id=cf2b2689ad7546168e08dcb4864a16b0) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:38.212555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7094b13-8800-49db-9e0a-7935a41e8fc3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:38.212555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7094b13-8800-49db-9e0a-7935a41e8fc3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:38.217618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 437/879] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:48:38 2022] GET /identity/v3/role_assignments?user.id=cf2b2689ad7546168e08dcb4864a16b0&scope.project.id=3352dfb8b28f41debbb48e69491e79b2&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:48:40.365558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33f246c-ca17-4e03-9314-e827ef49b023 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:40.366054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33f246c-ca17-4e03-9314-e827ef49b023 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:40.366054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33f246c-ca17-4e03-9314-e827ef49b023 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:40.367355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 443/880] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:40.372590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8629059-7cf2-40e2-b99a-3b91f9b06559 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:40.372951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8629059-7cf2-40e2-b99a-3b91f9b06559 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:40.373140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8629059-7cf2-40e2-b99a-3b91f9b06559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:40.391414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c8629059-7cf2-40e2-b99a-3b91f9b06559 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:40.392586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c8629059-7cf2-40e2-b99a-3b91f9b06559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:40.457294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 438/881] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:40.724453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33f246c-ca17-4e03-9314-e827ef49b023 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:40.777147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:40.830586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33f246c-ca17-4e03-9314-e827ef49b023 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:40.863792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3277d884-e0f7-4267-8dbe-1266a3468fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:40.864409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3277d884-e0f7-4267-8dbe-1266a3468fdd None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:40.865255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3277d884-e0f7-4267-8dbe-1266a3468fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:40.865673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3277d884-e0f7-4267-8dbe-1266a3468fdd None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:40.866353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3277d884-e0f7-4267-8dbe-1266a3468fdd None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:40.867723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3277d884-e0f7-4267-8dbe-1266a3468fdd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:40.944874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 444/882] 10.222.0.66 () {60 vars in 1138 bytes} [Fri Oct 28 19:48:40 2022] POST /identity/v3/domains => generated 267 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:43.099444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63edc4-cb34-4170-aac9-011e72ebbdcf None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:43.100030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63edc4-cb34-4170-aac9-011e72ebbdcf None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:43.100320 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63edc4-cb34-4170-aac9-011e72ebbdcf None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:43.101722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 439/883] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:43.107066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b85bf-7021-4dbb-b1d5-3d7cca926361 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:43.107492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b85bf-7021-4dbb-b1d5-3d7cca926361 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:43.107899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b85bf-7021-4dbb-b1d5-3d7cca926361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:43.126216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-825b85bf-7021-4dbb-b1d5-3d7cca926361 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:43.127375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-825b85bf-7021-4dbb-b1d5-3d7cca926361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:43.191233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 445/884] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:43.463213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63edc4-cb34-4170-aac9-011e72ebbdcf None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:43.561121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63edc4-cb34-4170-aac9-011e72ebbdcf None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:43.575319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:43.594053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab0613f-2a74-4214-ad48-e9abbb425d8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:43.594667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab0613f-2a74-4214-ad48-e9abbb425d8f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:43.594944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab0613f-2a74-4214-ad48-e9abbb425d8f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:43.595216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab0613f-2a74-4214-ad48-e9abbb425d8f None admin] PATH_INFO: `/v3/domains/c029f0bb95c44a9483946ca6269e95a5` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:43.606100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab0613f-2a74-4214-ad48-e9abbb425d8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=c029f0bb95c44a9483946ca6269e95a5)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:43.607531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab0613f-2a74-4214-ad48-e9abbb425d8f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:43.609071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 440/885] 10.222.0.66 () {56 vars in 1183 bytes} [Fri Oct 28 19:48:43 2022] GET /identity/v3/domains/c029f0bb95c44a9483946ca6269e95a5 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:43.615652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825b85bf-7021-4dbb-b1d5-3d7cca926361 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:43.656599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:43.711481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825b85bf-7021-4dbb-b1d5-3d7cca926361 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:43.745760 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d05bbd-53b8-4ad0-ae8d-e4f04eee37e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:43.746303 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d05bbd-53b8-4ad0-ae8d-e4f04eee37e3 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:43.746586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d05bbd-53b8-4ad0-ae8d-e4f04eee37e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:43.746863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d05bbd-53b8-4ad0-ae8d-e4f04eee37e3 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:43.747511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d05bbd-53b8-4ad0-ae8d-e4f04eee37e3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:43.748975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d05bbd-53b8-4ad0-ae8d-e4f04eee37e3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:43.846994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 446/886] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:48:43 2022] POST /identity/v3/projects => generated 361 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:46.010896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a970484b-497e-4ae3-b811-05a104531f7d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:46.011456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a970484b-497e-4ae3-b811-05a104531f7d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:46.011456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a970484b-497e-4ae3-b811-05a104531f7d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:46.012777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 441/887] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:46.018223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f1216-9be9-4596-897c-85f312071f99 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:46.018641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f1216-9be9-4596-897c-85f312071f99 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:46.018780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129f1216-9be9-4596-897c-85f312071f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:46.037854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-129f1216-9be9-4596-897c-85f312071f99 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:46.039031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-129f1216-9be9-4596-897c-85f312071f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:46.103200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 447/888] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:46.379693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a970484b-497e-4ae3-b811-05a104531f7d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:46.479567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a970484b-497e-4ae3-b811-05a104531f7d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:46.512110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df627a-34dd-454b-b615-019b1b4e65ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:46.512731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df627a-34dd-454b-b615-019b1b4e65ea None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:46.513016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df627a-34dd-454b-b615-019b1b4e65ea None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:46.513290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df627a-34dd-454b-b615-019b1b4e65ea None admin] PATH_INFO: `/v3/domains/c029f0bb95c44a9483946ca6269e95a5` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:46.514433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df627a-34dd-454b-b615-019b1b4e65ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=c029f0bb95c44a9483946ca6269e95a5)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:46.515863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df627a-34dd-454b-b615-019b1b4e65ea None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:46.517439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 442/889] 10.222.0.66 () {56 vars in 1183 bytes} [Fri Oct 28 19:48:46 2022] GET /identity/v3/domains/c029f0bb95c44a9483946ca6269e95a5 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:46.523738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129f1216-9be9-4596-897c-85f312071f99 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:46.548419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:46.622452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129f1216-9be9-4596-897c-85f312071f99 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:46.656465 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4ff56c-9ce1-40a1-a98f-b75641a06382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:46.657013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4ff56c-9ce1-40a1-a98f-b75641a06382 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:46.657300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4ff56c-9ce1-40a1-a98f-b75641a06382 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:46.657608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4ff56c-9ce1-40a1-a98f-b75641a06382 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:46.658294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4ff56c-9ce1-40a1-a98f-b75641a06382 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:46.659728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4ff56c-9ce1-40a1-a98f-b75641a06382 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:46.737261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 448/890] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:48:46 2022] POST /identity/v3/users => generated 311 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:48:48.921877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be7ccd-0753-40b5-8535-b5a269494737 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:48.922439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be7ccd-0753-40b5-8535-b5a269494737 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:48.922439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be7ccd-0753-40b5-8535-b5a269494737 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:48.923762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 443/891] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:48.929196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc03db4-75bf-4c45-99a3-e4c08f4c1f96 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:48.929642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc03db4-75bf-4c45-99a3-e4c08f4c1f96 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:48.929788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc03db4-75bf-4c45-99a3-e4c08f4c1f96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:48.948465 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0fc03db4-75bf-4c45-99a3-e4c08f4c1f96 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:48.949993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0fc03db4-75bf-4c45-99a3-e4c08f4c1f96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:49.015574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 449/892] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:49.282780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be7ccd-0753-40b5-8535-b5a269494737 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:49.325914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:49.383678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be7ccd-0753-40b5-8535-b5a269494737 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:49.415206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:49.415826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:49.416109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:49.416384 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:49.427726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:49.432523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:49.433729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:48:49.434982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 444/893] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:48:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:49.439826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc03db4-75bf-4c45-99a3-e4c08f4c1f96 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:49.451656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:49.536154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc03db4-75bf-4c45-99a3-e4c08f4c1f96 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:49.570124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:49.570725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:49.571007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:49.571307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:49.571960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:49.572279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:49.573671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:49.581585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 450/894] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:48:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:49.587247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:49.683565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e0b6db-3537-4c5f-b97a-ca21ab4f1e9b None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:49.685024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:49.716238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:49.716853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:49.717130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:49.717427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] PATH_INFO: `/v3/users/b26a36e483ec449983a34357115099b2` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:49.730542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] RBAC: Authorizing `identity:get_user(user_id=b26a36e483ec449983a34357115099b2)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:49.731962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:49.733534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 445/895] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:49 2022] GET /identity/v3/users/b26a36e483ec449983a34357115099b2 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:49.739186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:49.808021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:49.836202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dac81e-4d75-48f9-acd1-2b3726a6ec7a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:49.869178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c107b60-8c8c-4345-8648-cf97c0101f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:49.869829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c107b60-8c8c-4345-8648-cf97c0101f84 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:49.870113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c107b60-8c8c-4345-8648-cf97c0101f84 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:49.870387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c107b60-8c8c-4345-8648-cf97c0101f84 None admin] PATH_INFO: `/v3/projects/e363a038ae754c16b8cca76448339cf3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:49.871480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c107b60-8c8c-4345-8648-cf97c0101f84 None admin] RBAC: Authorizing `identity:get_project(project_id=e363a038ae754c16b8cca76448339cf3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:49.872900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c107b60-8c8c-4345-8648-cf97c0101f84 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:49.874561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 451/896] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:49 2022] GET /identity/v3/projects/e363a038ae754c16b8cca76448339cf3 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:49.880574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:49.976963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1247a2a3-a60b-44f2-a075-c101bba978cd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:50.008846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a780718c-355b-435b-bed3-cc8ea6ee8a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:50.009530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a780718c-355b-435b-bed3-cc8ea6ee8a5a None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:50.009813 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a780718c-355b-435b-bed3-cc8ea6ee8a5a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:50.010086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a780718c-355b-435b-bed3-cc8ea6ee8a5a None admin] PATH_INFO: `/v3/projects/e363a038ae754c16b8cca76448339cf3/users/b26a36e483ec449983a34357115099b2/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:50.011966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a780718c-355b-435b-bed3-cc8ea6ee8a5a None admin] RBAC: Authorizing `identity:create_grant(project_id=e363a038ae754c16b8cca76448339cf3, user_id=b26a36e483ec449983a34357115099b2, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:50.013433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a780718c-355b-435b-bed3-cc8ea6ee8a5a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:50.091593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 446/897] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:48:49 2022] PUT /identity/v3/projects/e363a038ae754c16b8cca76448339cf3/users/b26a36e483ec449983a34357115099b2/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:48:52.275392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6b7bab-0962-4a41-b752-832769ad8048 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:52.275929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6b7bab-0962-4a41-b752-832769ad8048 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:52.275929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6b7bab-0962-4a41-b752-832769ad8048 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:52.277289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 452/898] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:48:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:48:52.282509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d781e91-a3c5-42a6-a2f3-0c37a298c390 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:52.282911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d781e91-a3c5-42a6-a2f3-0c37a298c390 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:52.283017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d781e91-a3c5-42a6-a2f3-0c37a298c390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:52.302717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-9d781e91-a3c5-42a6-a2f3-0c37a298c390 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:48:52.303908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9d781e91-a3c5-42a6-a2f3-0c37a298c390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:48:52.318265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:52.370880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 447/899] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:48:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:48:52.640652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6b7bab-0962-4a41-b752-832769ad8048 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:52.741498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6b7bab-0962-4a41-b752-832769ad8048 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:52.775215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:52.775836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:52.776122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:52.776398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:52.783694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:52.787249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:48:52.792150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:48:52.793247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:48:52.794475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 453/900] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:48:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:48:52.799356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d781e91-a3c5-42a6-a2f3-0c37a298c390 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:52.899239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d781e91-a3c5-42a6-a2f3-0c37a298c390 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:52.934565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:52.935166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:52.935442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:52.935706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:52.936354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:52.936653 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:52.938035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:52.946050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 448/901] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:48:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:52.952158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:53.052132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e7394-f64f-48ce-96f4-33c7902dc411 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:53.085225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:53.085877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:53.086159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:53.086434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] PATH_INFO: `/v3/users/b26a36e483ec449983a34357115099b2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:53.087541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] RBAC: Authorizing `identity:get_user(user_id=b26a36e483ec449983a34357115099b2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:53.088964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:53.090570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 454/902] 10.222.0.66 () {56 vars in 1177 bytes} [Fri Oct 28 19:48:52 2022] GET /identity/v3/users/b26a36e483ec449983a34357115099b2 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:53.096329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:53.112071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:53.193062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f58509-3cb6-4d27-a477-31cccbb03944 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:53.225561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:53.226151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:53.226429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:53.226699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:53.290962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:53.292392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:53.305425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:53.386216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 449/903] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:48:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 19:48:53.392092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:53.456665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:53.488887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a570aa2-575e-4353-9e4f-e7f48dedb09b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:53.523333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c703e-cf61-4057-8127-2536fd3616f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:53.523947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c703e-cf61-4057-8127-2536fd3616f6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:53.524229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c703e-cf61-4057-8127-2536fd3616f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:53.524503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c703e-cf61-4057-8127-2536fd3616f6 None admin] PATH_INFO: `/v3/projects/e363a038ae754c16b8cca76448339cf3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:53.525613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2c703e-cf61-4057-8127-2536fd3616f6 None admin] RBAC: Authorizing `identity:get_project(project_id=e363a038ae754c16b8cca76448339cf3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:53.527028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2c703e-cf61-4057-8127-2536fd3616f6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:53.528614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 455/904] 10.222.0.66 () {56 vars in 1186 bytes} [Fri Oct 28 19:48:53 2022] GET /identity/v3/projects/e363a038ae754c16b8cca76448339cf3 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:48:53.534612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:48:53.632395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b93bf81-5d88-4faa-9914-15a1d99f3c43 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:48:53.645036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:48:53.666651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a8570-acc9-40fe-94d8-f950a4e8a601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:48:53.667250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a8570-acc9-40fe-94d8-f950a4e8a601 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:48:53.667534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a8570-acc9-40fe-94d8-f950a4e8a601 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:48:53.667811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a8570-acc9-40fe-94d8-f950a4e8a601 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:48:53.668527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a8570-acc9-40fe-94d8-f950a4e8a601 None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=e363a038ae754c16b8cca76448339cf3, user.id=b26a36e483ec449983a34357115099b2) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:48:53.668835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a8570-acc9-40fe-94d8-f950a4e8a601 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:48:53.670290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a8570-acc9-40fe-94d8-f950a4e8a601 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:48:53.675880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 450/905] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:48:53 2022] GET /identity/v3/role_assignments?user.id=b26a36e483ec449983a34357115099b2&scope.project.id=e363a038ae754c16b8cca76448339cf3&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:49:03.518658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=32 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:49:03.680479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=12 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:05.966209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cf301f-476a-4393-97b7-73740f70ae9d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:05.966798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cf301f-476a-4393-97b7-73740f70ae9d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:05.966798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cf301f-476a-4393-97b7-73740f70ae9d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:05.968091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 456/906] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:05.973310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d0aa6-fae8-4086-ab7d-9a3f84660ffe None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:05.973748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d0aa6-fae8-4086-ab7d-9a3f84660ffe None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:05.973899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d0aa6-fae8-4086-ab7d-9a3f84660ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:05.992865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-124d0aa6-fae8-4086-ab7d-9a3f84660ffe None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:05.994148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-124d0aa6-fae8-4086-ab7d-9a3f84660ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:06.062315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 451/907] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:06.334356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cf301f-476a-4393-97b7-73740f70ae9d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:06.434216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cf301f-476a-4393-97b7-73740f70ae9d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:06.468593 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:06.469229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:06.469543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:06.469820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:06.480566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:06.482190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:06.496712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:06.497938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:06.498966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:06.498966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:06.498966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:06.498966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 457/908] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:06.504101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124d0aa6-fae8-4086-ab7d-9a3f84660ffe None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:06.601734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124d0aa6-fae8-4086-ab7d-9a3f84660ffe None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:06.630519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:06.636833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f0085-fd0f-4e54-b7d9-7817246f257c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:06.636833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f0085-fd0f-4e54-b7d9-7817246f257c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:06.637085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f0085-fd0f-4e54-b7d9-7817246f257c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:06.637394 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f0085-fd0f-4e54-b7d9-7817246f257c None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:06.638033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f0085-fd0f-4e54-b7d9-7817246f257c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:06.638336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f0085-fd0f-4e54-b7d9-7817246f257c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:06.639702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f0085-fd0f-4e54-b7d9-7817246f257c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:06.650601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 452/909] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:06.656445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:06.710290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:06.756605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c13513-9510-43ad-a9b7-c828f1c1347f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:06.789232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a86c33f-f865-4e94-adf8-82aefffc82c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:06.790065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86c33f-f865-4e94-adf8-82aefffc82c4 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:06.790565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86c33f-f865-4e94-adf8-82aefffc82c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:06.791346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86c33f-f865-4e94-adf8-82aefffc82c4 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:06.792005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a86c33f-f865-4e94-adf8-82aefffc82c4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:06.794109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a86c33f-f865-4e94-adf8-82aefffc82c4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:06.918487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 458/910] 10.222.0.66 () {60 vars in 1132 bytes} [Fri Oct 28 19:50:06 2022] POST /identity/v3/users => generated 251 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:50:09.112478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc62637-51ec-4aaa-87b4-177aafad7520 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:09.113010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc62637-51ec-4aaa-87b4-177aafad7520 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:09.113116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc62637-51ec-4aaa-87b4-177aafad7520 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:09.114428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 453/911] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:09.119813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22b96b-fcbf-484c-888a-d5de542c9967 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:09.120208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22b96b-fcbf-484c-888a-d5de542c9967 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:09.121153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22b96b-fcbf-484c-888a-d5de542c9967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:09.139037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6a22b96b-fcbf-484c-888a-d5de542c9967 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:09.140223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6a22b96b-fcbf-484c-888a-d5de542c9967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:09.210025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 459/912] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:09.482952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc62637-51ec-4aaa-87b4-177aafad7520 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:09.600978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc62637-51ec-4aaa-87b4-177aafad7520 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:09.633969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:09.634600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:09.634899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:09.635171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:09.645511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:50:09.650316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:09.651489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:09.653502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 454/913] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:50:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:09.658184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a22b96b-fcbf-484c-888a-d5de542c9967 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:09.697907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:09.755402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a22b96b-fcbf-484c-888a-d5de542c9967 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:09.788846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:09.789486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:09.795170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:09.795170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:09.795170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:09.795170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:09.795170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:09.800896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 460/914] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:50:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:09.806950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:09.858465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:09.908941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dfd212-d9b0-4c8e-a9f3-2b94b0f9860e None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:09.943423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:09.944043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:09.944320 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:09.944588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:09.955426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:09.956876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:09.971228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:09.972368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:09.973253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:09.973253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:09.973253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:09.973253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 455/915] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:09.978301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:10.076629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:10.079655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269ac0d-78c2-4520-8cf1-307a1f402fba None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:10.112725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:10.113346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:10.113631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:10.113909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:10.114551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:10.114857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:10.116267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:10.128404 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 461/916] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:10.134164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:10.233780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d4e3fe-f7d3-428b-a596-a356a89df8ff None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:10.235334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:10.266720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:10.267341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:10.267622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:10.267897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:10.275049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:10.276485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:50:10.288235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:10.289334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:10.290529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 456/917] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:10 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:10.296280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:10.395284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce7295c-54da-437e-a293-1f8438446a94 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:10.430266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:10.430873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:10.431163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:10.431441 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:10.432160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:10.432467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:10.434420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:10.452659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 462/918] 10.222.0.66 () {56 vars in 1143 bytes} [Fri Oct 28 19:50:10 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:10.458520 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:10.565944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b89bf61-2115-4b7d-8024-6479f493d935 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:10.599452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:10.600076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:10.600449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:10.600729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:10.611871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:10.613396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:10.628245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:10.629438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:10.630714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:10.630714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:10.630714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:10.630714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 457/919] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:10.635181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:10.653736 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:10.734304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770ee43-e74a-46d6-8130-bd4f4401b68a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:10.767389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:10.767996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:10.768279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:10.768555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:10.769194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:10.769530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:10.770959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:10.781843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 463/920] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:10.787550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:10.803893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:10.886207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a317ed-d863-4586-bcc3-f3e75ef00c91 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:10.919956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:10.920559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:10.920839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:10.921117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:10.927625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:10.929077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:50:10.940413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:10.941753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:10.942842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 458/921] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:50:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:10.947856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:11.029090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:11.050897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294d4d07-9b52-4476-acca-8a417295294a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:11.084154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64e2034-341f-47a9-b18b-e8f659624d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:11.084758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64e2034-341f-47a9-b18b-e8f659624d11 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:11.085042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64e2034-341f-47a9-b18b-e8f659624d11 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:11.085344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64e2034-341f-47a9-b18b-e8f659624d11 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:11.086080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64e2034-341f-47a9-b18b-e8f659624d11 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:11.086607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64e2034-341f-47a9-b18b-e8f659624d11 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:11.088617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64e2034-341f-47a9-b18b-e8f659624d11 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:11.101524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 464/922] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:50:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:11.108485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:11.184898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:11.208869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b4f33c-ae1f-4ecf-9000-4d9b1f0d0954 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:11.241673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8b6680-3b57-4c0a-be4c-57682d978a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:11.242327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8b6680-3b57-4c0a-be4c-57682d978a6e None admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:11.242605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8b6680-3b57-4c0a-be4c-57682d978a6e None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:11.242876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8b6680-3b57-4c0a-be4c-57682d978a6e None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/5fbacc77af9b43f1a2c1cf15cd09e337/roles/57b3843b68f846d5905474ee762f7059` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:11.256785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8b6680-3b57-4c0a-be4c-57682d978a6e None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=5fbacc77af9b43f1a2c1cf15cd09e337, role_id=57b3843b68f846d5905474ee762f7059)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:11.258554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8b6680-3b57-4c0a-be4c-57682d978a6e None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:11.355242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 459/923] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:50:11 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/5fbacc77af9b43f1a2c1cf15cd09e337/roles/57b3843b68f846d5905474ee762f7059 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:50:13.545966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c65d35-1ed6-422d-97bf-d8d7ceb143d0 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:13.546691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c65d35-1ed6-422d-97bf-d8d7ceb143d0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:13.546691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c65d35-1ed6-422d-97bf-d8d7ceb143d0 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:13.548732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 465/924] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:13.555402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8d604-b6fe-496a-83f8-8c4bcb22b8e0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:13.555830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8d604-b6fe-496a-83f8-8c4bcb22b8e0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:13.555921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8d604-b6fe-496a-83f8-8c4bcb22b8e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:13.575030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-dbd8d604-b6fe-496a-83f8-8c4bcb22b8e0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:13.576216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-dbd8d604-b6fe-496a-83f8-8c4bcb22b8e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:13.640959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 460/925] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:13.911065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c65d35-1ed6-422d-97bf-d8d7ceb143d0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:14.012207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c65d35-1ed6-422d-97bf-d8d7ceb143d0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:14.047988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:14.048620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:14.048900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:14.049171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:14.056505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:14.060594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:50:14.065649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:14.067058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:14.067972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 466/926] 10.222.0.66 () {56 vars in 1102 bytes} [Fri Oct 28 19:50:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:14.073181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8d604-b6fe-496a-83f8-8c4bcb22b8e0 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:14.129076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:14.173972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8d604-b6fe-496a-83f8-8c4bcb22b8e0 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:14.208589 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:14.209193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:14.209519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:14.209788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:14.210465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] RBAC: Adding query filter params (name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:14.210773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:14.212185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:14.219779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 461/927] 10.222.0.66 () {56 vars in 1103 bytes} [Fri Oct 28 19:50:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:14.225722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:14.325812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32340664-8725-49aa-8d9c-df7d3756eeb1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:14.359518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:14.360143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:14.360425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:14.360700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:14.370604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:14.372340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:14.386771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:14.388231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:14.394566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:14.394566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:14.394566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:14.394566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 467/928] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:14.394808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:14.492824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c16af57-7944-494c-826a-47c844a821ad None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:14.508780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:14.526487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b93446-9361-4981-897c-a03078b801cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:14.527090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b93446-9361-4981-897c-a03078b801cd None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:14.527372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b93446-9361-4981-897c-a03078b801cd None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:14.527643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b93446-9361-4981-897c-a03078b801cd None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:14.528872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b93446-9361-4981-897c-a03078b801cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:14.529168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b93446-9361-4981-897c-a03078b801cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:14.530569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b93446-9361-4981-897c-a03078b801cd None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:14.541834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 462/929] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:14.548440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:14.598055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:14.645574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc3b723-ff19-4891-9e86-f3f43d64c419 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:14.679102 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:14.679724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:14.680001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:14.680273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:14.687613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:14.689033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:50:14.700217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:14.701388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:14.702430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 468/930] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:14.707248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b93446-9361-4981-897c-a03078b801cd None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:14.805131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b93446-9361-4981-897c-a03078b801cd None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:14.839178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:14.839776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:14.840060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:14.840336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:14.842098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:14.842407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:14.843843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:14.862192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 463/931] 10.222.0.66 () {56 vars in 1143 bytes} [Fri Oct 28 19:50:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:14.868004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:14.968147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c91af7-4881-4f06-8b81-1290e4978804 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:14.977951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:15.001625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:15.004624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:15.004624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:15.004624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:15.070591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:15.071972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:15.170947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 469/932] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:50:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 19:50:15.176817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:15.210461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:15.273230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d076613-85d5-4e53-89a6-dbf79960fc92 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:15.306829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:15.307451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:15.307730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:15.307996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:15.318678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:15.320186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:15.337734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:15.338877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:15.340067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:15.340067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:15.340067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:15.340067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 464/933] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:15.342295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:15.353715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:15.441373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48e0a5d-912b-4216-82d9-26c2843c26ac None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:15.480647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:15.481987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:15.482332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:15.482651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:15.483500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:15.483808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:15.485182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:15.496322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 470/934] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:15.502412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:15.592460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:15.607007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b303b-532e-43a5-aba1-162a9db166e7 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:15.639648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:15.640449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:15.640625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:15.640808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:15.647239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:15.648683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:50:15.660706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:15.668372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:15.670346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 465/935] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:50:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:15.671055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:15.738958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:15.769058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027acba-cafa-4f47-aac5-03837767aee3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:15.801515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f916cca8-c075-4eed-9e95-e3199994b161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:15.802124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f916cca8-c075-4eed-9e95-e3199994b161 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:15.802410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f916cca8-c075-4eed-9e95-e3199994b161 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:15.802861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f916cca8-c075-4eed-9e95-e3199994b161 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:15.803564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f916cca8-c075-4eed-9e95-e3199994b161 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:15.803866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f916cca8-c075-4eed-9e95-e3199994b161 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:15.805291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f916cca8-c075-4eed-9e95-e3199994b161 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:15.817746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 471/936] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:50:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:15.823822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:15.923306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85686bda-0fb7-4567-9bcc-e2ad51264b66 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:15.957835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e704083-0bce-4eb8-859e-3b858af56351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:15.958441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e704083-0bce-4eb8-859e-3b858af56351 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:15.958727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e704083-0bce-4eb8-859e-3b858af56351 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:15.959003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e704083-0bce-4eb8-859e-3b858af56351 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:15.959721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e704083-0bce-4eb8-859e-3b858af56351 None admin] RBAC: Adding query filter params (role.id=57b3843b68f846d5905474ee762f7059, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=5fbacc77af9b43f1a2c1cf15cd09e337) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:15.960031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e704083-0bce-4eb8-859e-3b858af56351 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:15.961530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e704083-0bce-4eb8-859e-3b858af56351 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:15.968077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 466/937] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:50:15 2022] GET /identity/v3/role_assignments?user.id=5fbacc77af9b43f1a2c1cf15cd09e337&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=57b3843b68f846d5905474ee762f7059 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:18.211807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9275b1-af32-4385-bf14-d5a53ea0289e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:18.212400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9275b1-af32-4385-bf14-d5a53ea0289e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:18.212506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9275b1-af32-4385-bf14-d5a53ea0289e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:18.213838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 472/938] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:18.219588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb17c898-8cd4-4cbd-94a6-38204c198271 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:18.220007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb17c898-8cd4-4cbd-94a6-38204c198271 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:18.220140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb17c898-8cd4-4cbd-94a6-38204c198271 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:18.238095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-cb17c898-8cd4-4cbd-94a6-38204c198271 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:18.239277 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-cb17c898-8cd4-4cbd-94a6-38204c198271 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:18.241969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:18.303596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 467/939] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:18.582373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9275b1-af32-4385-bf14-d5a53ea0289e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:18.684304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9275b1-af32-4385-bf14-d5a53ea0289e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:18.712962 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:18.718986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:18.719651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:18.719938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:18.720213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:18.729941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:50:18.735210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:18.748365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:18.749470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 473/940] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:50:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:18.750349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb17c898-8cd4-4cbd-94a6-38204c198271 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:18.840358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb17c898-8cd4-4cbd-94a6-38204c198271 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:18.876198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:18.880214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:18.880214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:18.880214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:18.880214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:18.880214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:18.880214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:18.888735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 468/941] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:50:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:18.894751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:18.994684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5069da-fe2e-4306-a134-ba4c808ccc11 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:19.029076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:19.029756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:19.030041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:19.030314 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:19.040327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:19.045495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:19.056091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:19.057243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:19.058584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:19.058584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:19.058584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:19.058584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 474/942] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:19.063685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:19.066553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:19.164474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56b402a-446a-4d4f-91d2-9dad6171a6c4 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:19.198283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:19.198876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:19.199179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:19.199447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:19.200672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:19.200966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:19.202359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:19.213665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 469/943] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:19.219525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:19.261895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:19.316478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ced7b96-0deb-44f8-980e-eebe7e184676 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:19.349765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:19.350395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:19.350679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:19.350952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:19.358394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:19.359836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:50:19.372234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:19.373229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:19.375477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 475/944] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:19 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:19.379401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:19.442700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:19.479271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383abcca-49ca-4c59-b6b7-f1c8d9e686f7 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:19.512021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:19.512656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:19.513104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:19.513408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:19.514125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:19.514429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:19.515875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:19.532445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 470/945] 10.222.0.66 () {56 vars in 1143 bytes} [Fri Oct 28 19:50:19 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:19.538300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:19.646034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae74fce-3fd1-42d2-9e43-9c3329a86e0b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:19.647561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:19.678738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:19.679373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:19.679661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:19.679936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:19.690913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:19.692593 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:19.707118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:19.708224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:19.709145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:19.709145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:19.709145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:19.709145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 476/946] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 19:50:19.713931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:19.813925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d154f08-63e4-434a-a7cb-0874cb9bad4c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:19.834728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:19.848616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:19.849222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:19.849531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:19.849804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:19.850436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:19.850737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:19.852099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:19.863432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 471/947] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:19.869393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:19.968702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fa55b-c6b9-4fef-86f7-e1ee3a4c0fc3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:20.001193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:20.001846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:20.002133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:20.002416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:20.009047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:20.011949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:50:20.024417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:20.029612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:20.030599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 477/948] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:50:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:20.032207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:20.131502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e886793-52c4-40ac-a698-fa4bc73990b2 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:20.164799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3eb840-7fcf-4eec-a7c4-c52d6b8c00d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:20.165421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3eb840-7fcf-4eec-a7c4-c52d6b8c00d9 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:20.165701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3eb840-7fcf-4eec-a7c4-c52d6b8c00d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:20.166043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3eb840-7fcf-4eec-a7c4-c52d6b8c00d9 None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:20.166724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3eb840-7fcf-4eec-a7c4-c52d6b8c00d9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:20.167026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3eb840-7fcf-4eec-a7c4-c52d6b8c00d9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:20.168442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3eb840-7fcf-4eec-a7c4-c52d6b8c00d9 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:20.180577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 472/949] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:50:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 19:50:20.186755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:20.204198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:20.286075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979fb1f0-23a0-49b4-84d9-0db047f64f68 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:20.319662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb16e16b-bc08-43b7-bd55-55fc219b342c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:20.320315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb16e16b-bc08-43b7-bd55-55fc219b342c None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:20.320605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb16e16b-bc08-43b7-bd55-55fc219b342c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:20.320880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb16e16b-bc08-43b7-bd55-55fc219b342c None admin] PATH_INFO: `/v3/projects/2399c000b584431197e605f2c47df1b0/users/5fbacc77af9b43f1a2c1cf15cd09e337/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:20.322788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb16e16b-bc08-43b7-bd55-55fc219b342c None admin] RBAC: Authorizing `identity:create_grant(project_id=2399c000b584431197e605f2c47df1b0, user_id=5fbacc77af9b43f1a2c1cf15cd09e337, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:20.324225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb16e16b-bc08-43b7-bd55-55fc219b342c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:20.408463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 478/950] 10.222.0.66 () {58 vars in 1439 bytes} [Fri Oct 28 19:50:20 2022] PUT /identity/v3/projects/2399c000b584431197e605f2c47df1b0/users/5fbacc77af9b43f1a2c1cf15cd09e337/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 19:50:22.567765 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a66c40-a4cf-4791-be3b-c7fe3a718ad4 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:22.568318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a66c40-a4cf-4791-be3b-c7fe3a718ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:22.568318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a66c40-a4cf-4791-be3b-c7fe3a718ad4 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:22.569688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 473/951] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:22.574880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1539c8d1-fc0c-43b9-a67b-4296aff75f7a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:22.575274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1539c8d1-fc0c-43b9-a67b-4296aff75f7a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:22.575445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1539c8d1-fc0c-43b9-a67b-4296aff75f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:22.595432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1539c8d1-fc0c-43b9-a67b-4296aff75f7a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:22.596615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1539c8d1-fc0c-43b9-a67b-4296aff75f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:22.664531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 479/952] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 19:50:22.944270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a66c40-a4cf-4791-be3b-c7fe3a718ad4 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:22.989402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:23.046728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a66c40-a4cf-4791-be3b-c7fe3a718ad4 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:23.079607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:23.080226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:23.080504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:23.080773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:23.090348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 19:50:23.095460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:23.096586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:23.097823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 474/953] 10.222.0.66 () {56 vars in 1096 bytes} [Fri Oct 28 19:50:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:23.102756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1539c8d1-fc0c-43b9-a67b-4296aff75f7a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:23.105863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:23.202503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1539c8d1-fc0c-43b9-a67b-4296aff75f7a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:23.235060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:23.235665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:23.235950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:23.236223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:23.236872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:23.237175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:23.238564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:23.246306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 480/954] 10.222.0.66 () {56 vars in 1099 bytes} [Fri Oct 28 19:50:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:23.252772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:23.349494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:23.352431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d2f0df-1113-441b-b2b2-69a106f7ddf3 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:23.387572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:23.388263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:23.388617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:23.388966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:23.399160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:23.400688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:23.415082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:23.421300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:23.422529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:23.422529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:23.422529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:23.422529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 475/955] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:23.422816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:23.486513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:23.519629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8d54ab-920c-41ab-a459-4040474bc5c2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:23.552074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:23.555819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:23.555819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:23.555819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:23.555819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:23.555819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:23.555819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:23.567058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 481/956] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:23.572838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:23.675558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c783dc-0716-4a45-8e46-e78cc9f20c6c None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:23.708514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:23.709179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:23.709484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:23.709759 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] PATH_INFO: `/v3/users/placement` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:23.717710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:23.719125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 19:50:23.730092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:23.731273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:23.733081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 476/957] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:23 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:23.737896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:23.835776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ac8ec-eb46-46ed-97e6-ba832e08f9c9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:23.864574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:23.873090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:23.873728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:23.874009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:23.874281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:23.875038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:23.875331 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:23.876742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:23.893634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 482/958] 10.222.0.66 () {56 vars in 1143 bytes} [Fri Oct 28 19:50:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:23.899435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:23.914816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:23.995651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5eb742-13a3-4b56-aeaf-0c85bf1c61ee None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:24.030834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:24.031455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:24.031739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:24.032015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:24.101016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:24.102534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:24.116079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:24.206207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 477/959] 10.222.0.66 () {58 vars in 1282 bytes} [Fri Oct 28 19:50:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 19:50:24.212186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:24.319048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c505bf-242b-46a7-bad1-f11db32e7f8c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:24.363527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:24.364165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:24.364491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:24.364757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:24.377623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:24.379080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 19:50:24.397351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:24.399504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 19:50:24.400445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 19:50:24.400445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 19:50:24.400445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:24.400445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 483/960] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 19:50:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:24.404567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:24.515854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08456709-534c-4ae6-bc0b-b0c15b27b322 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:24.529947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:24.552816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:24.553499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:24.553775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:24.554051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:24.554694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:24.554999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:24.556373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:24.570835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 478/961] 10.222.0.66 () {56 vars in 1109 bytes} [Fri Oct 28 19:50:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:24.580544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:24.631642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:24.718446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cafb147-4f4a-44e7-8cb9-cf23f4399ccc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:24.758651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:24.759347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:24.759558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:24.759831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:24.769025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:24.770584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 19:50:24.788784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:24.794986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 19:50:24.796250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 484/962] 10.222.0.66 () {56 vars in 1111 bytes} [Fri Oct 28 19:50:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:24.798250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:24.926690 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c90fe75-c3a9-4221-aeba-952b3b80aff7 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:24.971916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc5dc2b-4a33-4d01-9431-ae53a91e4e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:24.972868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5dc2b-4a33-4d01-9431-ae53a91e4e4f None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:24.973156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5dc2b-4a33-4d01-9431-ae53a91e4e4f None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:24.973466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5dc2b-4a33-4d01-9431-ae53a91e4e4f None admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:24.974167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc5dc2b-4a33-4d01-9431-ae53a91e4e4f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:24.974475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc5dc2b-4a33-4d01-9431-ae53a91e4e4f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:24.975922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc5dc2b-4a33-4d01-9431-ae53a91e4e4f None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:24.991862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 479/963] 10.222.0.66 () {56 vars in 1148 bytes} [Fri Oct 28 19:50:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:24.998511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:25.116759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:25.123012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb54cbb-b0c2-4420-a2d6-e3c371e250f1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:25.157382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c263226-6632-4f5e-b0d6-e98a8e6ed76b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:25.157998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c263226-6632-4f5e-b0d6-e98a8e6ed76b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:25.158283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c263226-6632-4f5e-b0d6-e98a8e6ed76b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:25.158561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c263226-6632-4f5e-b0d6-e98a8e6ed76b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:25.159283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c263226-6632-4f5e-b0d6-e98a8e6ed76b None admin] RBAC: Adding query filter params (role.id=e2631d01cdf5420a8210f6ff7e885fef, scope.project.id=2399c000b584431197e605f2c47df1b0, user.id=5fbacc77af9b43f1a2c1cf15cd09e337) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:25.159594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c263226-6632-4f5e-b0d6-e98a8e6ed76b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:25.161026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c263226-6632-4f5e-b0d6-e98a8e6ed76b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:25.166918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 485/964] 10.222.0.66 () {56 vars in 1374 bytes} [Fri Oct 28 19:50:24 2022] GET /identity/v3/role_assignments?user.id=5fbacc77af9b43f1a2c1cf15cd09e337&scope.project.id=2399c000b584431197e605f2c47df1b0&role.id=e2631d01cdf5420a8210f6ff7e885fef => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:27.358725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28506967-d92f-4e85-9c7c-9a84d24b662e None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:27.359231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28506967-d92f-4e85-9c7c-9a84d24b662e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:27.359231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28506967-d92f-4e85-9c7c-9a84d24b662e None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:27.360584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 480/965] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:27.365943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eca0979-1d95-430d-80ed-d9f3e1b9f1ed None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:27.366421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eca0979-1d95-430d-80ed-d9f3e1b9f1ed None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:27.366496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eca0979-1d95-430d-80ed-d9f3e1b9f1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:27.384226 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7eca0979-1d95-430d-80ed-d9f3e1b9f1ed None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:27.385426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7eca0979-1d95-430d-80ed-d9f3e1b9f1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:27.449739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 486/966] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:27.719558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28506967-d92f-4e85-9c7c-9a84d24b662e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:27.751155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:27.820813 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28506967-d92f-4e85-9c7c-9a84d24b662e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:27.855106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:27.855740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:27.856023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:27.856297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:27.856990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:27.858418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:27.867457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:27.868853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:27.869973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 481/967] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:50:27 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:27.874720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eca0979-1d95-430d-80ed-d9f3e1b9f1ed None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:27.962504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:27.972193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eca0979-1d95-430d-80ed-d9f3e1b9f1ed None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:28.004323 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394e9ccc-f5cd-4665-8c03-7343aded2137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:28.004937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394e9ccc-f5cd-4665-8c03-7343aded2137 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:28.005222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394e9ccc-f5cd-4665-8c03-7343aded2137 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:28.005542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394e9ccc-f5cd-4665-8c03-7343aded2137 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:28.006372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394e9ccc-f5cd-4665-8c03-7343aded2137 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:28.006679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394e9ccc-f5cd-4665-8c03-7343aded2137 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:28.008043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394e9ccc-f5cd-4665-8c03-7343aded2137 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:28.012936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 487/968] 10.222.0.66 () {56 vars in 1116 bytes} [Fri Oct 28 19:50:27 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:28.019600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:28.114606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:28.120244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70950856-5439-4b91-b36c-103c592c1128 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:28.154841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4e9cb3-30db-4561-a8ae-c9f0b538c088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:28.155453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e9cb3-30db-4561-a8ae-c9f0b538c088 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:28.155741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e9cb3-30db-4561-a8ae-c9f0b538c088 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:28.156021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e9cb3-30db-4561-a8ae-c9f0b538c088 None admin] PATH_INFO: `/v3/services` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:28.156665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e9cb3-30db-4561-a8ae-c9f0b538c088 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:28.156965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e9cb3-30db-4561-a8ae-c9f0b538c088 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:28.158382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e9cb3-30db-4561-a8ae-c9f0b538c088 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:28.162746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 482/969] 10.222.0.66 () {56 vars in 1116 bytes} [Fri Oct 28 19:50:28 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:30.281784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e7a8ea-4533-4154-af5f-5b5258d7d2a8 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:30.282308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e7a8ea-4533-4154-af5f-5b5258d7d2a8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:30.282308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e7a8ea-4533-4154-af5f-5b5258d7d2a8 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:30.283594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 488/970] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:30.288987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24818b1-e98b-4fb7-a794-04a48bdd8f90 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:30.289553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24818b1-e98b-4fb7-a794-04a48bdd8f90 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:30.289664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24818b1-e98b-4fb7-a794-04a48bdd8f90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:30.307620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d24818b1-e98b-4fb7-a794-04a48bdd8f90 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:30.308799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d24818b1-e98b-4fb7-a794-04a48bdd8f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:30.375354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 483/971] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:30.643549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e7a8ea-4533-4154-af5f-5b5258d7d2a8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:30.744118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e7a8ea-4533-4154-af5f-5b5258d7d2a8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:30.777366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93232f1-97b6-4caa-bb19-64142c7b707f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:30.778017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93232f1-97b6-4caa-bb19-64142c7b707f None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:30.778311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93232f1-97b6-4caa-bb19-64142c7b707f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:30.778585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93232f1-97b6-4caa-bb19-64142c7b707f None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:30.779540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93232f1-97b6-4caa-bb19-64142c7b707f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:30.780908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93232f1-97b6-4caa-bb19-64142c7b707f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:30.887788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 489/972] 10.222.0.66 () {60 vars in 1141 bytes} [Fri Oct 28 19:50:30 2022] POST /identity/v3/services => generated 246 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:50:33.112537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7980e35-6257-424c-9485-21f00eeb79f5 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:33.113062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7980e35-6257-424c-9485-21f00eeb79f5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:33.113167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7980e35-6257-424c-9485-21f00eeb79f5 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:33.114499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 484/973] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:33.119841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9e7b0-ed0a-41e8-9858-3dd8ff4e9215 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:33.120273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9e7b0-ed0a-41e8-9858-3dd8ff4e9215 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:33.120391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9e7b0-ed0a-41e8-9858-3dd8ff4e9215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:33.138511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-20b9e7b0-ed0a-41e8-9858-3dd8ff4e9215 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:33.139693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-20b9e7b0-ed0a-41e8-9858-3dd8ff4e9215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:33.148898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:33.210746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 490/974] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:33 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:33.481429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7980e35-6257-424c-9485-21f00eeb79f5 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:33.583397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:33.593846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7980e35-6257-424c-9485-21f00eeb79f5 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:33.631947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:33.632598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:33.632880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:33.633151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:33.633772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:33.635189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:33.644919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:33.646511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:33.647347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:33.649496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 485/975] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:50:33 2022] GET /identity/v3/services/placement => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:33.652292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b9e7b0-ed0a-41e8-9858-3dd8ff4e9215 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:33.761033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b9e7b0-ed0a-41e8-9858-3dd8ff4e9215 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:33.798893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:33.799521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:33.799802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:33.800075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:33.800710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] RBAC: Adding query filter params (name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:33.801013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:33.802446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:33.808355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 491/976] 10.222.0.66 () {56 vars in 1116 bytes} [Fri Oct 28 19:50:33 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:33.814885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:33.921101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8b9e8e-fee9-4153-b21e-2ce02e951841 None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:33.959946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9720ea5a-fb79-4fd5-9a72-09d2396db7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:33.960562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9720ea5a-fb79-4fd5-9a72-09d2396db7ac None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:33.960847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9720ea5a-fb79-4fd5-9a72-09d2396db7ac None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:33.961123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9720ea5a-fb79-4fd5-9a72-09d2396db7ac None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:33.961887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9720ea5a-fb79-4fd5-9a72-09d2396db7ac None admin] RBAC: Adding query filter params (interface=public, service_id=b977a719940b434085088409b3c2ac56, region_id=RegionOne) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:33.962188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9720ea5a-fb79-4fd5-9a72-09d2396db7ac None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:33.963564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9720ea5a-fb79-4fd5-9a72-09d2396db7ac None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:33.970004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 486/977] 10.222.0.66 () {56 vars in 1251 bytes} [Fri Oct 28 19:50:33 2022] GET /identity/v3/endpoints?service_id=b977a719940b434085088409b3c2ac56&interface=public®ion_id=RegionOne => generated 194 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:33.975862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:33.979774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:34.074497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138f13e-2d6f-4555-889b-24ecda9fa0ae None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:34.107411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfc4846-12cc-4e33-8425-1a1487606623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:34.108011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfc4846-12cc-4e33-8425-1a1487606623 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:34.108297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfc4846-12cc-4e33-8425-1a1487606623 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:34.108574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfc4846-12cc-4e33-8425-1a1487606623 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:34.109190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfc4846-12cc-4e33-8425-1a1487606623 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:34.110588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfc4846-12cc-4e33-8425-1a1487606623 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:34.117084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 492/978] 10.222.0.66 () {56 vars in 1087 bytes} [Fri Oct 28 19:50:33 2022] GET /identity/v3/services => generated 2191 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 19:50:36.260293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad75f41-fb07-45b5-9688-777c3c019693 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:36.260845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad75f41-fb07-45b5-9688-777c3c019693 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:36.260845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad75f41-fb07-45b5-9688-777c3c019693 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:36.262222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 487/979] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:50:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:50:36.267616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e6929-1f20-486b-90fd-42f1d06bab85 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:36.268045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e6929-1f20-486b-90fd-42f1d06bab85 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:36.268171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e6929-1f20-486b-90fd-42f1d06bab85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:36.286635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a69e6929-1f20-486b-90fd-42f1d06bab85 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:50:36.287738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a69e6929-1f20-486b-90fd-42f1d06bab85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:50:36.349556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:36.355017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 493/980] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:50:36 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 19:50:36.621202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad75f41-fb07-45b5-9688-777c3c019693 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:36.640620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:36.723364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad75f41-fb07-45b5-9688-777c3c019693 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:36.756221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:36.756847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:36.757126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:36.757428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] PATH_INFO: `/v3/services/placement` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:36.758002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:36.759369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 19:50:36.769341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 19:50:36.770549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 19:50:36.771569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 488/981] 10.222.0.66 () {56 vars in 1117 bytes} [Fri Oct 28 19:50:36 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 19:50:36.776479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69e6929-1f20-486b-90fd-42f1d06bab85 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:36.874886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69e6929-1f20-486b-90fd-42f1d06bab85 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:36.907442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2986b6d0-8875-4604-b085-b8c86dc7a6e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:36.908047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986b6d0-8875-4604-b085-b8c86dc7a6e7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:36.908325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986b6d0-8875-4604-b085-b8c86dc7a6e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:36.908596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986b6d0-8875-4604-b085-b8c86dc7a6e7 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:36.909211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2986b6d0-8875-4604-b085-b8c86dc7a6e7 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 19:50:36.909545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2986b6d0-8875-4604-b085-b8c86dc7a6e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:36.910899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2986b6d0-8875-4604-b085-b8c86dc7a6e7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:36.915842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 494/982] 10.222.0.66 () {56 vars in 1116 bytes} [Fri Oct 28 19:50:36 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 19:50:36.921795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:50:37.002154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:37.020123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37e7017-7756-424f-83bb-cc0a0823808a None admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:50:37.053686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b259cf5-1546-4370-a425-c1cd1b8c7ce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:50:37.054299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b259cf5-1546-4370-a425-c1cd1b8c7ce3 None admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:50:37.054583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b259cf5-1546-4370-a425-c1cd1b8c7ce3 None admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:50:37.054858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b259cf5-1546-4370-a425-c1cd1b8c7ce3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:50:37.055521 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b259cf5-1546-4370-a425-c1cd1b8c7ce3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:50:37.056897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b259cf5-1546-4370-a425-c1cd1b8c7ce3 None admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:50:37.155269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 489/983] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 19:50:36 2022] POST /identity/v3/endpoints => generated 331 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 19:50:46.912865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=69 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:50:47.150663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:01.220893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb679350-3858-45b2-b30f-759d262abe21 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:01.221370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb679350-3858-45b2-b30f-759d262abe21 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:01.221462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb679350-3858-45b2-b30f-759d262abe21 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:01.228594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152d58d-3193-4a99-b797-f000c3947f06 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:01.228976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152d58d-3193-4a99-b797-f000c3947f06 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:01.229142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152d58d-3193-4a99-b797-f000c3947f06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:01.233831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 495/984] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:01 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:01.308213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0152d58d-3193-4a99-b797-f000c3947f06 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:01.309458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0152d58d-3193-4a99-b797-f000c3947f06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:01.433692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 490/985] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:01.722401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d17555b-fcd4-4a60-a536-50d373ace934 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:01.722691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d17555b-fcd4-4a60-a536-50d373ace934 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:01.723215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d17555b-fcd4-4a60-a536-50d373ace934 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:01.724693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 496/986] 10.222.0.66 () {54 vars in 965 bytes} [Fri Oct 28 19:57:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:01.731825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737b6235-6e81-4c08-954b-10dbfe223b89 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:01.732209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737b6235-6e81-4c08-954b-10dbfe223b89 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:01.732601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737b6235-6e81-4c08-954b-10dbfe223b89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:01.818923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-737b6235-6e81-4c08-954b-10dbfe223b89 None None] MFA Rules not processed for user `91639dc527ae47e892164544572cbec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:01.820728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-737b6235-6e81-4c08-954b-10dbfe223b89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:01.843258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:01.902335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 491/987] 10.222.0.66 () {58 vars in 1062 bytes} [Fri Oct 28 19:57:01 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:01.912629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d17555b-fcd4-4a60-a536-50d373ace934 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:01.915109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3d17555b-fcd4-4a60-a536-50d373ace934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:02.040819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d17555b-fcd4-4a60-a536-50d373ace934 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:02.132101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a01f22-ea08-4f46-bee4-004a16103c00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:02.132884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a01f22-ea08-4f46-bee4-004a16103c00 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:02.133170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a01f22-ea08-4f46-bee4-004a16103c00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:02.133476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a01f22-ea08-4f46-bee4-004a16103c00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:02.135217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e7a01f22-ea08-4f46-bee4-004a16103c00 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:02.202226 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:02.263453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a01f22-ea08-4f46-bee4-004a16103c00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:02.264912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a01f22-ea08-4f46-bee4-004a16103c00 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:02.286213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 497/988] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:04.827782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d706c0-9221-4baf-ac33-5f88e2bf0921 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:04.828375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d706c0-9221-4baf-ac33-5f88e2bf0921 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:04.828375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d706c0-9221-4baf-ac33-5f88e2bf0921 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:04.829712 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 492/989] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:04.836102 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2efea8-0cbc-47ce-94ae-9bfdc0faca0e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:04.836682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2efea8-0cbc-47ce-94ae-9bfdc0faca0e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:04.836682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2efea8-0cbc-47ce-94ae-9bfdc0faca0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:04.927352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3f2efea8-0cbc-47ce-94ae-9bfdc0faca0e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:04.929356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3f2efea8-0cbc-47ce-94ae-9bfdc0faca0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:05.035506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 498/990] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:05.322673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d706c0-9221-4baf-ac33-5f88e2bf0921 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:05.324217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-96d706c0-9221-4baf-ac33-5f88e2bf0921 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:05.428176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d706c0-9221-4baf-ac33-5f88e2bf0921 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:05.499755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:05.501953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ab989-9045-4ee8-952a-9e792b526f35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:05.502557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ab989-9045-4ee8-952a-9e792b526f35 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:05.502837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ab989-9045-4ee8-952a-9e792b526f35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:05.503109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ab989-9045-4ee8-952a-9e792b526f35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:05.504685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1f0ab989-9045-4ee8-952a-9e792b526f35 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:05.611559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ab989-9045-4ee8-952a-9e792b526f35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:05.612974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ab989-9045-4ee8-952a-9e792b526f35 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:05.632183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 493/991] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:08.263326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eac302-64e6-4ad3-a62d-1dd6c34ba7fd None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:08.263888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eac302-64e6-4ad3-a62d-1dd6c34ba7fd None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:08.263888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eac302-64e6-4ad3-a62d-1dd6c34ba7fd None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:08.265216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 499/992] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:08.270397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a43a15-ec41-4397-9ea4-af9ddbcb86f8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:08.270821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a43a15-ec41-4397-9ea4-af9ddbcb86f8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:08.270949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a43a15-ec41-4397-9ea4-af9ddbcb86f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:08.336720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-17a43a15-ec41-4397-9ea4-af9ddbcb86f8 None None] MFA Rules not processed for user `2e21a1848e62493eb1c08ca9a1bdbcc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:08.337885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-17a43a15-ec41-4397-9ea4-af9ddbcb86f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:08.389055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:08.433773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 494/993] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:08 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:08.717546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0eac302-64e6-4ad3-a62d-1dd6c34ba7fd None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:08.719081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f0eac302-64e6-4ad3-a62d-1dd6c34ba7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:08.729503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:08.822560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0eac302-64e6-4ad3-a62d-1dd6c34ba7fd None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:08.897045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29ce7eb-4531-4365-bebc-f04ba9dace34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:08.897683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29ce7eb-4531-4365-bebc-f04ba9dace34 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:08.897968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29ce7eb-4531-4365-bebc-f04ba9dace34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:08.898244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29ce7eb-4531-4365-bebc-f04ba9dace34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:08.899830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c29ce7eb-4531-4365-bebc-f04ba9dace34 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:08.962501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:09.016965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29ce7eb-4531-4365-bebc-f04ba9dace34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:09.018446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29ce7eb-4531-4365-bebc-f04ba9dace34 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:09.037156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 500/994] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:08 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:12.010046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73de15da-7353-4bd8-8a07-bcca2a6300ab None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:12.010579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73de15da-7353-4bd8-8a07-bcca2a6300ab None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:12.010579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73de15da-7353-4bd8-8a07-bcca2a6300ab None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:12.011958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 495/995] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:12.017396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22832c1-972b-40f4-8c4a-2bdd92d3978f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:12.017807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22832c1-972b-40f4-8c4a-2bdd92d3978f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:12.017959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22832c1-972b-40f4-8c4a-2bdd92d3978f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:12.083797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c22832c1-972b-40f4-8c4a-2bdd92d3978f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:12.085604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c22832c1-972b-40f4-8c4a-2bdd92d3978f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:12.180753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 501/996] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:12 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:12.484095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e21f42-abca-4108-b049-c518afdf4657 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:12.484373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e21f42-abca-4108-b049-c518afdf4657 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:12.484645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e21f42-abca-4108-b049-c518afdf4657 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:12.486026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 496/997] 10.222.0.66 () {54 vars in 965 bytes} [Fri Oct 28 19:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:12.492167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f857533-accb-4155-96c8-01910a81c6f9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:12.492453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f857533-accb-4155-96c8-01910a81c6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:12.492729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f857533-accb-4155-96c8-01910a81c6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:12.514362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:12.567634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7f857533-accb-4155-96c8-01910a81c6f9 None None] MFA Rules not processed for user `91639dc527ae47e892164544572cbec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:12.569135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7f857533-accb-4155-96c8-01910a81c6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:12.643657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 502/998] 10.222.0.66 () {58 vars in 1062 bytes} [Fri Oct 28 19:57:12 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:12.653028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e21f42-abca-4108-b049-c518afdf4657 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:12.654591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f9e21f42-abca-4108-b049-c518afdf4657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:12.755447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e21f42-abca-4108-b049-c518afdf4657 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:12.828214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb05f8b-4fcf-43eb-a530-5ffe99d0ef1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:12.828818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb05f8b-4fcf-43eb-a530-5ffe99d0ef1f service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:12.829100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb05f8b-4fcf-43eb-a530-5ffe99d0ef1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:12.829400 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb05f8b-4fcf-43eb-a530-5ffe99d0ef1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:12.830988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-adb05f8b-4fcf-43eb-a530-5ffe99d0ef1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:12.838633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:12.938970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb05f8b-4fcf-43eb-a530-5ffe99d0ef1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:12.940395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb05f8b-4fcf-43eb-a530-5ffe99d0ef1f service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:12.958927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 497/999] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:16.603162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0eed1-fc00-432f-8884-5168eb18bd48 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:16.603694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0eed1-fc00-432f-8884-5168eb18bd48 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:16.603694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0eed1-fc00-432f-8884-5168eb18bd48 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:16.605044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 503/1000] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:16.610430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb9b41-2f27-401e-9445-83e4425cadfb None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:16.610880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb9b41-2f27-401e-9445-83e4425cadfb None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:16.611084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb9b41-2f27-401e-9445-83e4425cadfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:16.680073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0efb9b41-2f27-401e-9445-83e4425cadfb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:16.681261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0efb9b41-2f27-401e-9445-83e4425cadfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:16.743521 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:16.785873 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 498/1001] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:17.067842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0eed1-fc00-432f-8884-5168eb18bd48 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:17.070338 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b9c0eed1-fc00-432f-8884-5168eb18bd48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:17.173447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0eed1-fc00-432f-8884-5168eb18bd48 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:17.186379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:17.248133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302d9f66-14f6-401e-88aa-40bbe9897be9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:17.248745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302d9f66-14f6-401e-88aa-40bbe9897be9 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:17.249019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302d9f66-14f6-401e-88aa-40bbe9897be9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:17.249481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302d9f66-14f6-401e-88aa-40bbe9897be9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:17.251050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-302d9f66-14f6-401e-88aa-40bbe9897be9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:17.361146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302d9f66-14f6-401e-88aa-40bbe9897be9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:17.362604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302d9f66-14f6-401e-88aa-40bbe9897be9 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:17.381867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 504/1002] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:19.595536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95b6370-1690-4f23-abf9-b5b2f7812aff None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:19.596095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95b6370-1690-4f23-abf9-b5b2f7812aff None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:19.596095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95b6370-1690-4f23-abf9-b5b2f7812aff None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:19.597449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 499/1003] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:19.602765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57af8a8-ab49-46d7-9d3d-cb0c2c465509 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:19.603216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57af8a8-ab49-46d7-9d3d-cb0c2c465509 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:19.603329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57af8a8-ab49-46d7-9d3d-cb0c2c465509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:19.656759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:19.674693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b57af8a8-ab49-46d7-9d3d-cb0c2c465509 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:19.675919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b57af8a8-ab49-46d7-9d3d-cb0c2c465509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:19.775230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 505/1004] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:20.062530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95b6370-1690-4f23-abf9-b5b2f7812aff None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:20.064054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d95b6370-1690-4f23-abf9-b5b2f7812aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:20.169688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95b6370-1690-4f23-abf9-b5b2f7812aff None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:20.244827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3390df-cafa-42f7-8fd5-84485dc733ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:20.245465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3390df-cafa-42f7-8fd5-84485dc733ff service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:20.245749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3390df-cafa-42f7-8fd5-84485dc733ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:20.246020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3390df-cafa-42f7-8fd5-84485dc733ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:20.247615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fc3390df-cafa-42f7-8fd5-84485dc733ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:20.264618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:20.357979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3390df-cafa-42f7-8fd5-84485dc733ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:20.359417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3390df-cafa-42f7-8fd5-84485dc733ff service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:20.378784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 500/1005] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 19:57:22.612507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397f1931-dfc9-4683-9d9b-93aac406daec None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:22.613047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397f1931-dfc9-4683-9d9b-93aac406daec None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:22.613047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397f1931-dfc9-4683-9d9b-93aac406daec None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:22.614454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 506/1006] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:22.620031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74913f27-4fa5-4692-baa6-2f129d8f13d6 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:22.620437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74913f27-4fa5-4692-baa6-2f129d8f13d6 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:22.620586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74913f27-4fa5-4692-baa6-2f129d8f13d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:22.688631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-74913f27-4fa5-4692-baa6-2f129d8f13d6 None None] MFA Rules not processed for user `2e21a1848e62493eb1c08ca9a1bdbcc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:22.689875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-74913f27-4fa5-4692-baa6-2f129d8f13d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:22.758993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:22.791310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 501/1007] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:22 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:23.081520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397f1931-dfc9-4683-9d9b-93aac406daec None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:23.083074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-397f1931-dfc9-4683-9d9b-93aac406daec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:23.195895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397f1931-dfc9-4683-9d9b-93aac406daec None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:23.232703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:23.294632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a952b50-2086-464d-8219-ecb3b9ede1a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:23.295143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a952b50-2086-464d-8219-ecb3b9ede1a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:23.295546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a952b50-2086-464d-8219-ecb3b9ede1a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:23.295831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a952b50-2086-464d-8219-ecb3b9ede1a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:23.297502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6a952b50-2086-464d-8219-ecb3b9ede1a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:23.421839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a952b50-2086-464d-8219-ecb3b9ede1a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:23.423297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a952b50-2086-464d-8219-ecb3b9ede1a4 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:23.444236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 507/1008] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:23 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:26.187229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b19e-3e2a-4baf-bb43-d39da0c3ba21 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:26.187816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b19e-3e2a-4baf-bb43-d39da0c3ba21 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:26.187816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b19e-3e2a-4baf-bb43-d39da0c3ba21 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:26.189133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 502/1009] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:26.195067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ac1a18-40a3-4908-bf9b-5851ba9aa73c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:26.195481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ac1a18-40a3-4908-bf9b-5851ba9aa73c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:26.195609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ac1a18-40a3-4908-bf9b-5851ba9aa73c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:26.258326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:26.260966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e3ac1a18-40a3-4908-bf9b-5851ba9aa73c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:26.262193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e3ac1a18-40a3-4908-bf9b-5851ba9aa73c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:26.358880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 508/1010] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:57:26.651491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb1b19e-3e2a-4baf-bb43-d39da0c3ba21 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:26.653064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5bb1b19e-3e2a-4baf-bb43-d39da0c3ba21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:26.758330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb1b19e-3e2a-4baf-bb43-d39da0c3ba21 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:26.832138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa34b30e-d10c-4b05-9824-f482df205909 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:26.832746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa34b30e-d10c-4b05-9824-f482df205909 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:26.833027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa34b30e-d10c-4b05-9824-f482df205909 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:26.833299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa34b30e-d10c-4b05-9824-f482df205909 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:26.836088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-aa34b30e-d10c-4b05-9824-f482df205909 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:26.863687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:26.947701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa34b30e-d10c-4b05-9824-f482df205909 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:26.949155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa34b30e-d10c-4b05-9824-f482df205909 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:26.968018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 503/1011] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:29.163217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ac2f91-b884-4626-8397-371f590f346e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:29.163777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ac2f91-b884-4626-8397-371f590f346e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:29.163777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ac2f91-b884-4626-8397-371f590f346e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:29.165116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 509/1012] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:29.170631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91f0a1a-cd43-44d9-b9bd-7d54ad9bbde2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:29.171081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91f0a1a-cd43-44d9-b9bd-7d54ad9bbde2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:29.171261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91f0a1a-cd43-44d9-b9bd-7d54ad9bbde2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:29.241853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e91f0a1a-cd43-44d9-b9bd-7d54ad9bbde2 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:29.243069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e91f0a1a-cd43-44d9-b9bd-7d54ad9bbde2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:29.331354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:29.359691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 504/1013] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:29.684298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ac2f91-b884-4626-8397-371f590f346e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:29.685851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-89ac2f91-b884-4626-8397-371f590f346e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:29.799571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ac2f91-b884-4626-8397-371f590f346e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:29.838217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:29.890862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e61a4b-7ac6-4d76-8cc8-ee9ebdd9c39f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:29.891488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e61a4b-7ac6-4d76-8cc8-ee9ebdd9c39f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:29.891772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e61a4b-7ac6-4d76-8cc8-ee9ebdd9c39f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:29.892050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e61a4b-7ac6-4d76-8cc8-ee9ebdd9c39f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:29.893725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e5e61a4b-7ac6-4d76-8cc8-ee9ebdd9c39f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:30.023481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e61a4b-7ac6-4d76-8cc8-ee9ebdd9c39f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:30.024985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e61a4b-7ac6-4d76-8cc8-ee9ebdd9c39f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:30.047373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 510/1014] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:32.832451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1322b6-5128-4152-876c-d9540ec757a3 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:32.833015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1322b6-5128-4152-876c-d9540ec757a3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:32.833015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1322b6-5128-4152-876c-d9540ec757a3 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:32.834643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 505/1015] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:32.840121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69508210-af0d-40bd-8a7e-dfac2150c376 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:32.840530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69508210-af0d-40bd-8a7e-dfac2150c376 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:32.840668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69508210-af0d-40bd-8a7e-dfac2150c376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:32.907039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-69508210-af0d-40bd-8a7e-dfac2150c376 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:32.908238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-69508210-af0d-40bd-8a7e-dfac2150c376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:32.917104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:33.005767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 511/1016] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:32 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:57:33.304431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1322b6-5128-4152-876c-d9540ec757a3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:33.306008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8c1322b6-5128-4152-876c-d9540ec757a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:33.413498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1322b6-5128-4152-876c-d9540ec757a3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:33.537417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000c4304-334f-4975-b9e0-c05899d0a2c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:33.538029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c4304-334f-4975-b9e0-c05899d0a2c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:33.538311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c4304-334f-4975-b9e0-c05899d0a2c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:33.538585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c4304-334f-4975-b9e0-c05899d0a2c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:33.540180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-000c4304-334f-4975-b9e0-c05899d0a2c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:33.576293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:33.649438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000c4304-334f-4975-b9e0-c05899d0a2c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:33.650873 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000c4304-334f-4975-b9e0-c05899d0a2c9 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:33.669412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 506/1017] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:33 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:37.979958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf31c6c-4eba-47b6-92ec-5ca87350783c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:37.980515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf31c6c-4eba-47b6-92ec-5ca87350783c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:37.980515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf31c6c-4eba-47b6-92ec-5ca87350783c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:37.981884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 512/1018] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:37.987078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8fb9b3-5184-4211-8563-41443c34e5da None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:37.987503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8fb9b3-5184-4211-8563-41443c34e5da None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:37.987669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8fb9b3-5184-4211-8563-41443c34e5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:38.058223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-4d8fb9b3-5184-4211-8563-41443c34e5da None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:38.059446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4d8fb9b3-5184-4211-8563-41443c34e5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:38.143758 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:38.155347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 507/1019] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:38.441179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf31c6c-4eba-47b6-92ec-5ca87350783c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:38.443594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8bf31c6c-4eba-47b6-92ec-5ca87350783c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:38.546978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf31c6c-4eba-47b6-92ec-5ca87350783c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:38.586797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:38.623677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196a50c3-69ad-43e4-aa9c-94695864166f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:38.624297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196a50c3-69ad-43e4-aa9c-94695864166f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:38.624585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196a50c3-69ad-43e4-aa9c-94695864166f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:38.624864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196a50c3-69ad-43e4-aa9c-94695864166f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:38.626491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-196a50c3-69ad-43e4-aa9c-94695864166f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:38.737782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196a50c3-69ad-43e4-aa9c-94695864166f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:38.739232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196a50c3-69ad-43e4-aa9c-94695864166f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:38.759592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 513/1020] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:41.842418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a60f55-28e3-4683-b1f8-9f7833302a8f None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:41.842905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a60f55-28e3-4683-b1f8-9f7833302a8f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:41.842996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a60f55-28e3-4683-b1f8-9f7833302a8f None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:41.845158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 508/1021] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:41.850450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80abc7d-e5ce-4341-8604-327202a2b89e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:41.850909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80abc7d-e5ce-4341-8604-327202a2b89e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:41.851011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80abc7d-e5ce-4341-8604-327202a2b89e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:41.920794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b80abc7d-e5ce-4341-8604-327202a2b89e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:41.921942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b80abc7d-e5ce-4341-8604-327202a2b89e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:41.941297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:42.020330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 514/1022] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:41 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:42.306133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a60f55-28e3-4683-b1f8-9f7833302a8f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:42.307735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-80a60f55-28e3-4683-b1f8-9f7833302a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:42.415105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a60f55-28e3-4683-b1f8-9f7833302a8f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:42.498945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feca81bd-d21e-4f9d-923d-a1ef72e01168 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:42.500075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feca81bd-d21e-4f9d-923d-a1ef72e01168 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:42.500379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feca81bd-d21e-4f9d-923d-a1ef72e01168 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:42.500657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feca81bd-d21e-4f9d-923d-a1ef72e01168 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:42.502481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-feca81bd-d21e-4f9d-923d-a1ef72e01168 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:42.553404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:42.627067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feca81bd-d21e-4f9d-923d-a1ef72e01168 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:42.628698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feca81bd-d21e-4f9d-923d-a1ef72e01168 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:42.657971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 509/1023] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:47.032794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d7065-7fd0-4980-8c9e-2044f1993d19 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:47.033241 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d7065-7fd0-4980-8c9e-2044f1993d19 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:47.033491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d7065-7fd0-4980-8c9e-2044f1993d19 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:47.034714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 515/1024] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:57:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:47.040854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc99655-1e67-422e-8dbd-09d2ad1bec59 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:47.041383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc99655-1e67-422e-8dbd-09d2ad1bec59 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:47.041472 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc99655-1e67-422e-8dbd-09d2ad1bec59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:47.114072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5cc99655-1e67-422e-8dbd-09d2ad1bec59 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:47.115585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5cc99655-1e67-422e-8dbd-09d2ad1bec59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:47.216071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:47.223367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 510/1025] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:57:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:47.512172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d7065-7fd0-4980-8c9e-2044f1993d19 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:57:47.513792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0c4d7065-7fd0-4980-8c9e-2044f1993d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:47.623326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d7065-7fd0-4980-8c9e-2044f1993d19 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:57:47.670443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:47.695903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1fdd86-6e0c-4e2e-a34c-be18d42991e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:57:47.696505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1fdd86-6e0c-4e2e-a34c-be18d42991e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:47.696781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1fdd86-6e0c-4e2e-a34c-be18d42991e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:47.697051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1fdd86-6e0c-4e2e-a34c-be18d42991e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:47.698652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fa1fdd86-6e0c-4e2e-a34c-be18d42991e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:47.805899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1fdd86-6e0c-4e2e-a34c-be18d42991e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:57:47.807320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1fdd86-6e0c-4e2e-a34c-be18d42991e4 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:57:47.826154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 516/1026] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:57:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:57:52.747534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702b92d-088b-43ec-b7db-bcce0b6a61c2 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:52.748034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702b92d-088b-43ec-b7db-bcce0b6a61c2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:52.748034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702b92d-088b-43ec-b7db-bcce0b6a61c2 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:52.753430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 511/1027] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 19:57:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:52.758110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9bd94-af4e-497d-a742-d8784f77799d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:52.758110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9bd94-af4e-497d-a742-d8784f77799d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:52.758110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9bd94-af4e-497d-a742-d8784f77799d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:52.846751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7ed9bd94-af4e-497d-a742-d8784f77799d None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:52.849456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7ed9bd94-af4e-497d-a742-d8784f77799d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:52.896545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:53.038527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 517/1028] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 19:57:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:57:57.321095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b589aa74-1875-4ec3-97da-d2bb97c1e69f None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:57.331568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b589aa74-1875-4ec3-97da-d2bb97c1e69f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:57.331568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b589aa74-1875-4ec3-97da-d2bb97c1e69f None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:57.331568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 512/1029] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 19:57:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:57.331858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab048d3-a0b5-45e8-9c51-52ece95dac93 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:57.331858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab048d3-a0b5-45e8-9c51-52ece95dac93 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:57.331858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab048d3-a0b5-45e8-9c51-52ece95dac93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:57.415852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4ab048d3-a0b5-45e8-9c51-52ece95dac93 None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:57.417163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4ab048d3-a0b5-45e8-9c51-52ece95dac93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:57.499770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:57:57.541084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 518/1030] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 19:57:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:57:58.821897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f5a2f-645e-4b98-b390-fe85b140119b None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:58.822453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f5a2f-645e-4b98-b390-fe85b140119b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:58.822453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f5a2f-645e-4b98-b390-fe85b140119b None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:58.823783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 513/1031] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 19:57:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:57:58.828809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a16c7-7526-44e9-aa82-4b505478ba48 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:57:58.829275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a16c7-7526-44e9-aa82-4b505478ba48 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:57:58.829557 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a16c7-7526-44e9-aa82-4b505478ba48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:57:58.909649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2c6a16c7-7526-44e9-aa82-4b505478ba48 None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:57:58.910910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2c6a16c7-7526-44e9-aa82-4b505478ba48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:57:59.020153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 519/1032] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 19:57:58 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:58:02.594145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd4d6e4-a420-4624-b303-92916f3e4221 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:02.594641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd4d6e4-a420-4624-b303-92916f3e4221 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:02.594641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd4d6e4-a420-4624-b303-92916f3e4221 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:02.596361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 514/1033] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:02.602690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a8a6c-0df7-44ae-8fa7-e51d1097193a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:02.603136 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a8a6c-0df7-44ae-8fa7-e51d1097193a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:02.603232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a8a6c-0df7-44ae-8fa7-e51d1097193a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:02.644153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:02.679178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f54a8a6c-0df7-44ae-8fa7-e51d1097193a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:02.680442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f54a8a6c-0df7-44ae-8fa7-e51d1097193a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:02.789368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 520/1034] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:03.086597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5214fc-58ce-4bc8-b561-4eb76de67826 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:03.087178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5214fc-58ce-4bc8-b561-4eb76de67826 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:03.087658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5214fc-58ce-4bc8-b561-4eb76de67826 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:03.090136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 515/1035] 10.222.0.66 () {54 vars in 961 bytes} [Fri Oct 28 19:58:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:03.099414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394b133-5ef7-4ecb-81a3-7cf503a03b8c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:03.099962 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394b133-5ef7-4ecb-81a3-7cf503a03b8c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:03.100443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394b133-5ef7-4ecb-81a3-7cf503a03b8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:03.187457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0394b133-5ef7-4ecb-81a3-7cf503a03b8c None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:03.188962 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0394b133-5ef7-4ecb-81a3-7cf503a03b8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:03.270458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:03.346738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 521/1036] 10.222.0.66 () {58 vars in 1058 bytes} [Fri Oct 28 19:58:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:58:03.355983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5214fc-58ce-4bc8-b561-4eb76de67826 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:03.357772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6b5214fc-58ce-4bc8-b561-4eb76de67826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:03.526979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5214fc-58ce-4bc8-b561-4eb76de67826 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:03.604683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:03.650043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b805db-2a20-4006-a701-cd448583cf91 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:03.650700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b805db-2a20-4006-a701-cd448583cf91 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:03.651044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b805db-2a20-4006-a701-cd448583cf91 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:03.651323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b805db-2a20-4006-a701-cd448583cf91 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:03.653275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b2b805db-2a20-4006-a701-cd448583cf91 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:03.757887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b805db-2a20-4006-a701-cd448583cf91 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:03.759796 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b805db-2a20-4006-a701-cd448583cf91 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:03.780623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 516/1037] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:06.194492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71292ce2-0fe8-4ea4-bf0c-282e19498b65 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:06.195172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71292ce2-0fe8-4ea4-bf0c-282e19498b65 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:06.195172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71292ce2-0fe8-4ea4-bf0c-282e19498b65 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:06.201551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c5bff-6ec4-4dda-a908-717157262f13 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:06.202154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c5bff-6ec4-4dda-a908-717157262f13 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:06.202154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c5bff-6ec4-4dda-a908-717157262f13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:06.205746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 522/1038] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:06 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:06.275280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-fc2c5bff-6ec4-4dda-a908-717157262f13 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:06.276515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fc2c5bff-6ec4-4dda-a908-717157262f13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:06.317157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:06.382804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 517/1039] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:58:06.667026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71292ce2-0fe8-4ea4-bf0c-282e19498b65 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:06.668536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-71292ce2-0fe8-4ea4-bf0c-282e19498b65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:06.851029 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71292ce2-0fe8-4ea4-bf0c-282e19498b65 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:06.862345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:06.978477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05d3203-53ef-44a1-9967-11051a5b09f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:06.979090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05d3203-53ef-44a1-9967-11051a5b09f0 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:06.979518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05d3203-53ef-44a1-9967-11051a5b09f0 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:06.979792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05d3203-53ef-44a1-9967-11051a5b09f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:06.981427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e05d3203-53ef-44a1-9967-11051a5b09f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:07.078724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05d3203-53ef-44a1-9967-11051a5b09f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:07.080328 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05d3203-53ef-44a1-9967-11051a5b09f0 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:07.100611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 523/1040] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:09.810244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cdaf00-a986-4295-b67f-4e176a5b380a None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:09.810780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cdaf00-a986-4295-b67f-4e176a5b380a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:09.810887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cdaf00-a986-4295-b67f-4e176a5b380a None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:09.813112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 518/1041] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:09.818146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd60b30-cabc-4755-8cf9-45b90b5f6c36 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:09.818638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd60b30-cabc-4755-8cf9-45b90b5f6c36 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:09.819007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd60b30-cabc-4755-8cf9-45b90b5f6c36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:09.835306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:09.888352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3cd60b30-cabc-4755-8cf9-45b90b5f6c36 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:09.889633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3cd60b30-cabc-4755-8cf9-45b90b5f6c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:10.005733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 524/1042] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:10.287122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739378af-a9eb-45fc-a219-864486d62e51 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:10.287576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739378af-a9eb-45fc-a219-864486d62e51 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:10.287956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739378af-a9eb-45fc-a219-864486d62e51 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:10.290046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 519/1043] 10.222.0.66 () {54 vars in 961 bytes} [Fri Oct 28 19:58:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:10.295595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20c555-8f6e-495f-9c83-d2bbc36a8a2c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:10.295894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20c555-8f6e-495f-9c83-d2bbc36a8a2c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:10.296181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20c555-8f6e-495f-9c83-d2bbc36a8a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:10.381877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fc20c555-8f6e-495f-9c83-d2bbc36a8a2c None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:10.383158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fc20c555-8f6e-495f-9c83-d2bbc36a8a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:10.421919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:10.510795 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 525/1044] 10.222.0.66 () {58 vars in 1058 bytes} [Fri Oct 28 19:58:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:10.518851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739378af-a9eb-45fc-a219-864486d62e51 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:10.521738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-739378af-a9eb-45fc-a219-864486d62e51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:10.700452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739378af-a9eb-45fc-a219-864486d62e51 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:10.724303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:10.807921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a78fb-0e7f-4db9-8a45-81265e5f8d87 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:10.808403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a78fb-0e7f-4db9-8a45-81265e5f8d87 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:10.808794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a78fb-0e7f-4db9-8a45-81265e5f8d87 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:10.811074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 526/1045] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:58:10.817518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba38cd-d887-4fd3-a71c-78048fed27e8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:10.817518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba38cd-d887-4fd3-a71c-78048fed27e8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:10.818944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba38cd-d887-4fd3-a71c-78048fed27e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:10.822816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e4c1b-18d2-4a68-9ff6-2f9148d1b46d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:10.823469 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e4c1b-18d2-4a68-9ff6-2f9148d1b46d service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:10.823705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e4c1b-18d2-4a68-9ff6-2f9148d1b46d service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:10.823978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e4c1b-18d2-4a68-9ff6-2f9148d1b46d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:10.825924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6c5e4c1b-18d2-4a68-9ff6-2f9148d1b46d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:10.892634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d8ba38cd-d887-4fd3-a71c-78048fed27e8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:10.893891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d8ba38cd-d887-4fd3-a71c-78048fed27e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:10.931313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e4c1b-18d2-4a68-9ff6-2f9148d1b46d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:10.932963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e4c1b-18d2-4a68-9ff6-2f9148d1b46d service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:10.956437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 520/1046] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:11.002434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:11.012995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 527/1047] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:11.358952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453e5d1-1869-4f10-ad5f-d74707ae041d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:11.359238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453e5d1-1869-4f10-ad5f-d74707ae041d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:11.359512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453e5d1-1869-4f10-ad5f-d74707ae041d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:11.360863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 521/1048] 10.222.0.66 () {54 vars in 963 bytes} [Fri Oct 28 19:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:11.366588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc528d-a93e-4707-a367-c2639855b671 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:11.366987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc528d-a93e-4707-a367-c2639855b671 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:11.367311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc528d-a93e-4707-a367-c2639855b671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:11.461229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8bfc528d-a93e-4707-a367-c2639855b671 None None] MFA Rules not processed for user `183a4f23782143b0b234d7800b409452`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:11.462473 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8bfc528d-a93e-4707-a367-c2639855b671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:11.545990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 528/1049] 10.222.0.66 () {58 vars in 1060 bytes} [Fri Oct 28 19:58:11 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:11.554159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453e5d1-1869-4f10-ad5f-d74707ae041d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:11.555798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2453e5d1-1869-4f10-ad5f-d74707ae041d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:11.588234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:11.668332 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453e5d1-1869-4f10-ad5f-d74707ae041d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:11.756192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6247c64c-f806-4db4-8bd0-b62a2fec8aed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:11.756820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6247c64c-f806-4db4-8bd0-b62a2fec8aed service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:11.757105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6247c64c-f806-4db4-8bd0-b62a2fec8aed service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:11.757411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6247c64c-f806-4db4-8bd0-b62a2fec8aed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:11.759062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6247c64c-f806-4db4-8bd0-b62a2fec8aed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:11.841402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:11.877799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6247c64c-f806-4db4-8bd0-b62a2fec8aed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:11.879255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6247c64c-f806-4db4-8bd0-b62a2fec8aed service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:11.904358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 522/1050] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 19:58:13.413821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f8b57-9129-42bd-b660-e4513ff58dcc None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:13.414230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f8b57-9129-42bd-b660-e4513ff58dcc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:13.414230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f8b57-9129-42bd-b660-e4513ff58dcc None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:13.417679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 529/1051] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:13.421027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eff25-1abe-4fab-b018-86064ff5d121 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:13.421897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eff25-1abe-4fab-b018-86064ff5d121 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:13.422178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eff25-1abe-4fab-b018-86064ff5d121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:13.493814 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-377eff25-1abe-4fab-b018-86064ff5d121 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:13.495042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-377eff25-1abe-4fab-b018-86064ff5d121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:13.626057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 523/1052] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:58:13.918952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f8b57-9129-42bd-b660-e4513ff58dcc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:13.920550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f36f8b57-9129-42bd-b660-e4513ff58dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:14.013502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:14.077353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f8b57-9129-42bd-b660-e4513ff58dcc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:14.200467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faae33d-9f29-4586-ab56-e5afefcd3614 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:14.201082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faae33d-9f29-4586-ab56-e5afefcd3614 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:14.201386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faae33d-9f29-4586-ab56-e5afefcd3614 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:14.201666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faae33d-9f29-4586-ab56-e5afefcd3614 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:14.203246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7faae33d-9f29-4586-ab56-e5afefcd3614 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:14.254704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:14.299996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faae33d-9f29-4586-ab56-e5afefcd3614 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:14.302129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faae33d-9f29-4586-ab56-e5afefcd3614 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:14.322122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 530/1053] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:16.680892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7367a5-c592-4dce-821b-68ac58dbc394 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:16.681474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7367a5-c592-4dce-821b-68ac58dbc394 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:16.681756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7367a5-c592-4dce-821b-68ac58dbc394 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:16.683130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 524/1054] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:16.688629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b03d1f-3856-425b-a5b6-0c6d909b414b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:16.689043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b03d1f-3856-425b-a5b6-0c6d909b414b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:16.689538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b03d1f-3856-425b-a5b6-0c6d909b414b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:16.761875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f2b03d1f-3856-425b-a5b6-0c6d909b414b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:16.763254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f2b03d1f-3856-425b-a5b6-0c6d909b414b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:16.870412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 531/1055] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:17.177944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7367a5-c592-4dce-821b-68ac58dbc394 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:17.179428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9a7367a5-c592-4dce-821b-68ac58dbc394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:17.219855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:17.339510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7367a5-c592-4dce-821b-68ac58dbc394 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:17.457436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:17.463076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a64580-ed91-4cdf-b509-b436dbf5fc04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:17.463696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a64580-ed91-4cdf-b509-b436dbf5fc04 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:17.463975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a64580-ed91-4cdf-b509-b436dbf5fc04 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:17.464288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a64580-ed91-4cdf-b509-b436dbf5fc04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:17.465947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-70a64580-ed91-4cdf-b509-b436dbf5fc04 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:17.563753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a64580-ed91-4cdf-b509-b436dbf5fc04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:17.565259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a64580-ed91-4cdf-b509-b436dbf5fc04 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:17.590304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 525/1056] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:19.976990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad77c8e4-ac53-44a0-8525-c83f4f0d28f2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:19.977592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad77c8e4-ac53-44a0-8525-c83f4f0d28f2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:19.977778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad77c8e4-ac53-44a0-8525-c83f4f0d28f2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:19.979624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 532/1057] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:19.985613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a378d4-50ac-4368-a8a0-879792ea7702 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:19.986050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a378d4-50ac-4368-a8a0-879792ea7702 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:19.986332 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a378d4-50ac-4368-a8a0-879792ea7702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:20.072784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-45a378d4-50ac-4368-a8a0-879792ea7702 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:20.074336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-45a378d4-50ac-4368-a8a0-879792ea7702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:20.148490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:20.181294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 526/1058] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:20.479361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad77c8e4-ac53-44a0-8525-c83f4f0d28f2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:20.481362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ad77c8e4-ac53-44a0-8525-c83f4f0d28f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:20.518291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:20.660997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad77c8e4-ac53-44a0-8525-c83f4f0d28f2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:20.764768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:20.781805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f509051-e13e-4654-9af4-3fcb5c5f693f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:20.782426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f509051-e13e-4654-9af4-3fcb5c5f693f service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:20.782777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f509051-e13e-4654-9af4-3fcb5c5f693f service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:20.783060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f509051-e13e-4654-9af4-3fcb5c5f693f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:20.784667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2f509051-e13e-4654-9af4-3fcb5c5f693f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:20.885803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f509051-e13e-4654-9af4-3fcb5c5f693f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:20.887394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f509051-e13e-4654-9af4-3fcb5c5f693f service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:20.907795 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 533/1059] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:23.853978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91150e7a-b2ba-4f0e-93c8-8d654078655d None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:23.854510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91150e7a-b2ba-4f0e-93c8-8d654078655d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:23.854510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91150e7a-b2ba-4f0e-93c8-8d654078655d None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:23.855814 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 527/1060] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:23.861371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0471607-f47c-4b4f-8c34-952c010eb4e5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:23.861933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0471607-f47c-4b4f-8c34-952c010eb4e5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:23.861933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0471607-f47c-4b4f-8c34-952c010eb4e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:23.930332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c0471607-f47c-4b4f-8c34-952c010eb4e5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:23.931546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c0471607-f47c-4b4f-8c34-952c010eb4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:23.993204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:24.047974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 534/1061] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:24.334816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91150e7a-b2ba-4f0e-93c8-8d654078655d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:24.336337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-91150e7a-b2ba-4f0e-93c8-8d654078655d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:24.508440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91150e7a-b2ba-4f0e-93c8-8d654078655d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:24.565378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:24.629543 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe4daad-fa56-4e1d-8e0f-854be93df893 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:24.629993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe4daad-fa56-4e1d-8e0f-854be93df893 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:24.630269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe4daad-fa56-4e1d-8e0f-854be93df893 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:24.630547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe4daad-fa56-4e1d-8e0f-854be93df893 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:24.632194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5fe4daad-fa56-4e1d-8e0f-854be93df893 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:24.735778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe4daad-fa56-4e1d-8e0f-854be93df893 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:24.737494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe4daad-fa56-4e1d-8e0f-854be93df893 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:24.758731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 528/1062] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:27.114198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4fb9bb-ca54-42ae-a0e2-ea7301c03e3d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:27.114729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4fb9bb-ca54-42ae-a0e2-ea7301c03e3d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:27.114825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4fb9bb-ca54-42ae-a0e2-ea7301c03e3d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:27.116114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 535/1063] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:27.121590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eab857-c3e3-488e-a1b8-490b4d2ad7c5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:27.121994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eab857-c3e3-488e-a1b8-490b4d2ad7c5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:27.122115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eab857-c3e3-488e-a1b8-490b4d2ad7c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:27.189939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-66eab857-c3e3-488e-a1b8-490b4d2ad7c5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:27.191747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-66eab857-c3e3-488e-a1b8-490b4d2ad7c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:27.200620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:27.293202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 529/1064] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:27.555294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a233a1-b926-42d5-bdb1-9ed83ad9588c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:27.555641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a233a1-b926-42d5-bdb1-9ed83ad9588c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:27.555765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a233a1-b926-42d5-bdb1-9ed83ad9588c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:27.558255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 536/1065] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:27.564073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e6eafc-32d9-42cf-8192-cfe2eb6de704 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:27.564412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e6eafc-32d9-42cf-8192-cfe2eb6de704 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:27.564687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e6eafc-32d9-42cf-8192-cfe2eb6de704 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:27.588629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a233a1-b926-42d5-bdb1-9ed83ad9588c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:27.590248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-24a233a1-b926-42d5-bdb1-9ed83ad9588c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:27.653443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-16e6eafc-32d9-42cf-8192-cfe2eb6de704 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:27.654810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-16e6eafc-32d9-42cf-8192-cfe2eb6de704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:27.751499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:27.757437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a233a1-b926-42d5-bdb1-9ed83ad9588c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:27.763683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 530/1066] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:27.807680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:27.874477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0726b5e9-b186-4c73-b470-403a89d606cf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:27.875389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0726b5e9-b186-4c73-b470-403a89d606cf service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:27.875671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0726b5e9-b186-4c73-b470-403a89d606cf service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:27.875943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0726b5e9-b186-4c73-b470-403a89d606cf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:27.877784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0726b5e9-b186-4c73-b470-403a89d606cf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:27.973758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0726b5e9-b186-4c73-b470-403a89d606cf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:27.975343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0726b5e9-b186-4c73-b470-403a89d606cf service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:27.993960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 537/1067] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:28.069475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf36d2c-fee1-42f6-b4aa-77f6a9f1ba49 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:28.069697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf36d2c-fee1-42f6-b4aa-77f6a9f1ba49 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:28.069990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf36d2c-fee1-42f6-b4aa-77f6a9f1ba49 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:28.071371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 531/1068] 10.222.0.66 () {54 vars in 962 bytes} [Fri Oct 28 19:58:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:28.076896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9cb59-62cd-4667-b0c6-9a1f817dbfbb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:28.077184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9cb59-62cd-4667-b0c6-9a1f817dbfbb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:28.077495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9cb59-62cd-4667-b0c6-9a1f817dbfbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:28.140534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:28.160263 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-25b9cb59-62cd-4667-b0c6-9a1f817dbfbb None None] MFA Rules not processed for user `a7fae443ad92439aadd2f94ad3bca594`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:28.161568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-25b9cb59-62cd-4667-b0c6-9a1f817dbfbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:28.247629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 538/1069] 10.222.0.66 () {58 vars in 1059 bytes} [Fri Oct 28 19:58:28 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:28.256768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf36d2c-fee1-42f6-b4aa-77f6a9f1ba49 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:28.258344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-faf36d2c-fee1-42f6-b4aa-77f6a9f1ba49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:28.374230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf36d2c-fee1-42f6-b4aa-77f6a9f1ba49 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:28.448165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751ef111-7b96-4976-9c52-511fc75c57a0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7fae443ad92439aadd2f94ad3bca594', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:28.448778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751ef111-7b96-4976-9c52-511fc75c57a0 service glance] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:28.449060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751ef111-7b96-4976-9c52-511fc75c57a0 service glance] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:28.449353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751ef111-7b96-4976-9c52-511fc75c57a0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:28.450943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-751ef111-7b96-4976-9c52-511fc75c57a0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:28.495202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:28.562922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751ef111-7b96-4976-9c52-511fc75c57a0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:28.564365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751ef111-7b96-4976-9c52-511fc75c57a0 service glance] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:28.584406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 532/1070] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:29.480915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac1206-0fee-4747-ad5d-0f7abd86c954 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:29.481483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac1206-0fee-4747-ad5d-0f7abd86c954 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:29.481603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac1206-0fee-4747-ad5d-0f7abd86c954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:29.549824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-71ac1206-0fee-4747-ad5d-0f7abd86c954 None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:29.551763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-71ac1206-0fee-4747-ad5d-0f7abd86c954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:29.637906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 539/1071] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 19:58:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:29.681105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013e5693-d294-47ba-8783-777bc0bbccbc None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:29.681609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013e5693-d294-47ba-8783-777bc0bbccbc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:29.681814 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013e5693-d294-47ba-8783-777bc0bbccbc None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:29.683242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 533/1072] 10.222.0.66 () {54 vars in 943 bytes} [Fri Oct 28 19:58:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:58:29.689460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119323ba-3fc6-4c47-afaa-9c3e52b6122a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:29.689834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119323ba-3fc6-4c47-afaa-9c3e52b6122a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:29.690144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119323ba-3fc6-4c47-afaa-9c3e52b6122a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:29.701218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:29.768182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-119323ba-3fc6-4c47-afaa-9c3e52b6122a None None] MFA Rules not processed for user `0df958d5d645473c9c0403bf9b4ee846`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:29.770019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-119323ba-3fc6-4c47-afaa-9c3e52b6122a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:29.877981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 540/1073] 10.222.0.66 () {58 vars in 1040 bytes} [Fri Oct 28 19:58:29 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:29.887490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013e5693-d294-47ba-8783-777bc0bbccbc None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:29.889049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-013e5693-d294-47ba-8783-777bc0bbccbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:30.031258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013e5693-d294-47ba-8783-777bc0bbccbc None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:30.045361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:30.138253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788c29c0-892c-4c80-bdef-e1a32564a1db service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:30.138869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c29c0-892c-4c80-bdef-e1a32564a1db service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:30.139200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c29c0-892c-4c80-bdef-e1a32564a1db service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:30.139418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c29c0-892c-4c80-bdef-e1a32564a1db service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:30.141037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-788c29c0-892c-4c80-bdef-e1a32564a1db service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:30.226209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788c29c0-892c-4c80-bdef-e1a32564a1db service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:30.227802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788c29c0-892c-4c80-bdef-e1a32564a1db service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:30.235777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 534/1074] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 19:58:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 19:58:30.394879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f410db5-3cbf-4619-98e0-1ceca8a3eaa8 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:30.395124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f410db5-3cbf-4619-98e0-1ceca8a3eaa8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:30.395410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f410db5-3cbf-4619-98e0-1ceca8a3eaa8 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:30.396776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 541/1075] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:58:30.402057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d2450-45b7-4f41-925a-3966c137b82e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:30.402343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d2450-45b7-4f41-925a-3966c137b82e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:30.402622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d2450-45b7-4f41-925a-3966c137b82e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:30.466567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:30.475382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-fa1d2450-45b7-4f41-925a-3966c137b82e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:30.476594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fa1d2450-45b7-4f41-925a-3966c137b82e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:30.600578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 535/1076] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:30.900195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f410db5-3cbf-4619-98e0-1ceca8a3eaa8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:30.901748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0f410db5-3cbf-4619-98e0-1ceca8a3eaa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:30.983130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:31.071041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f410db5-3cbf-4619-98e0-1ceca8a3eaa8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:31.212082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9063def1-d67e-4a20-ac10-22391ae0bf07 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:31.213152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9063def1-d67e-4a20-ac10-22391ae0bf07 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:31.213689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9063def1-d67e-4a20-ac10-22391ae0bf07 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:31.214475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9063def1-d67e-4a20-ac10-22391ae0bf07 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:31.217103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9063def1-d67e-4a20-ac10-22391ae0bf07 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:31.262448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:31.335830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9063def1-d67e-4a20-ac10-22391ae0bf07 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:31.337885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9063def1-d67e-4a20-ac10-22391ae0bf07 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:31.364868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 542/1077] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:33.853266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27f9caa-be60-4c0a-93e7-2d50c11de1cc None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:33.853874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27f9caa-be60-4c0a-93e7-2d50c11de1cc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:33.853874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27f9caa-be60-4c0a-93e7-2d50c11de1cc None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:33.855453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 536/1078] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:33.860595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a30ccd-f5b8-4953-b555-728afc32c1b8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:33.861052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a30ccd-f5b8-4953-b555-728afc32c1b8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:33.861141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a30ccd-f5b8-4953-b555-728afc32c1b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:33.931104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-00a30ccd-f5b8-4953-b555-728afc32c1b8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:33.932316 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-00a30ccd-f5b8-4953-b555-728afc32c1b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:34.044615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 543/1079] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:34.341441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27f9caa-be60-4c0a-93e7-2d50c11de1cc None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:34.342932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b27f9caa-be60-4c0a-93e7-2d50c11de1cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:34.480238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:34.509495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27f9caa-be60-4c0a-93e7-2d50c11de1cc None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:34.640734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78341987-9f0b-4eb3-aa8f-8e3155641918 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:34.641473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78341987-9f0b-4eb3-aa8f-8e3155641918 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:34.641833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78341987-9f0b-4eb3-aa8f-8e3155641918 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:34.642186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78341987-9f0b-4eb3-aa8f-8e3155641918 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:34.643832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-78341987-9f0b-4eb3-aa8f-8e3155641918 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:34.744146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:34.747574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78341987-9f0b-4eb3-aa8f-8e3155641918 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:34.749246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78341987-9f0b-4eb3-aa8f-8e3155641918 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:34.782397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 537/1080] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:37.111660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e639f72-86ed-48ba-9117-b21dd4ac51c1 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:37.112210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e639f72-86ed-48ba-9117-b21dd4ac51c1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:37.112210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e639f72-86ed-48ba-9117-b21dd4ac51c1 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:37.114533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 544/1081] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:37.119684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e5a65-82eb-4282-81be-a380802d7ece None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:37.120103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e5a65-82eb-4282-81be-a380802d7ece None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:37.120233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e5a65-82eb-4282-81be-a380802d7ece None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:37.191901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-149e5a65-82eb-4282-81be-a380802d7ece None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:37.193201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-149e5a65-82eb-4282-81be-a380802d7ece None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:37.301581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 538/1082] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:37.592293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e639f72-86ed-48ba-9117-b21dd4ac51c1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:37.593851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0e639f72-86ed-48ba-9117-b21dd4ac51c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:37.598161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:37.763947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e639f72-86ed-48ba-9117-b21dd4ac51c1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:37.867477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:37.906742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1822a165-c5b3-4740-9a37-c544e1ef57d6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:37.907394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822a165-c5b3-4740-9a37-c544e1ef57d6 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:37.907678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822a165-c5b3-4740-9a37-c544e1ef57d6 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:37.907955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822a165-c5b3-4740-9a37-c544e1ef57d6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:37.910155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1822a165-c5b3-4740-9a37-c544e1ef57d6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:38.011857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1822a165-c5b3-4740-9a37-c544e1ef57d6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:38.013546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1822a165-c5b3-4740-9a37-c544e1ef57d6 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:38.036717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 545/1083] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:40.476799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bf86eb-db7a-4191-a390-440bf1725e52 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:40.476799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bf86eb-db7a-4191-a390-440bf1725e52 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:40.476799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bf86eb-db7a-4191-a390-440bf1725e52 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:40.476799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 539/1084] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 19:58:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:40.481019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8e953a-f4be-411c-aba6-8aa6da4a9c7e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:40.481460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8e953a-f4be-411c-aba6-8aa6da4a9c7e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:40.481754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8e953a-f4be-411c-aba6-8aa6da4a9c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:40.555500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9d8e953a-f4be-411c-aba6-8aa6da4a9c7e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:40.555500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9d8e953a-f4be-411c-aba6-8aa6da4a9c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:40.599908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:40.663563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 546/1085] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 19:58:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:40.955244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bf86eb-db7a-4191-a390-440bf1725e52 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:40.958379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-50bf86eb-db7a-4191-a390-440bf1725e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:41.039378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:41.132448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bf86eb-db7a-4191-a390-440bf1725e52 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:41.259609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b1f59-6e43-411d-80f5-9dd589a0b4c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:41.260344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b1f59-6e43-411d-80f5-9dd589a0b4c9 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:41.260631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b1f59-6e43-411d-80f5-9dd589a0b4c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:41.260905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b1f59-6e43-411d-80f5-9dd589a0b4c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:41.264024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6d9b1f59-6e43-411d-80f5-9dd589a0b4c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:41.295526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:41.368518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9b1f59-6e43-411d-80f5-9dd589a0b4c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:41.370244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9b1f59-6e43-411d-80f5-9dd589a0b4c9 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:41.399675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 540/1086] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:58:50.661979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=25 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:51.386824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=34 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:55.127571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc68b7-6902-4de8-b717-f53282689a52 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:55.128157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc68b7-6902-4de8-b717-f53282689a52 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:55.128266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc68b7-6902-4de8-b717-f53282689a52 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:55.129656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 547/1087] 10.222.0.66 () {54 vars in 918 bytes} [Fri Oct 28 19:58:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:55.134789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a86f04b-3a74-4a03-baf7-0440bf74466f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:55.147975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a86f04b-3a74-4a03-baf7-0440bf74466f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:55.147975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a86f04b-3a74-4a03-baf7-0440bf74466f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:55.212643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3a86f04b-3a74-4a03-baf7-0440bf74466f None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:55.213863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3a86f04b-3a74-4a03-baf7-0440bf74466f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:55.319100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 541/1088] 10.222.0.66 () {58 vars in 1015 bytes} [Fri Oct 28 19:58:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:55.333159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320d884a-1555-4540-95ef-f725e9334b45 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:55.333486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320d884a-1555-4540-95ef-f725e9334b45 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:55.333763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320d884a-1555-4540-95ef-f725e9334b45 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:55.335073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 548/1089] 10.222.0.66 () {54 vars in 918 bytes} [Fri Oct 28 19:58:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:55.339510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ed095-30ea-4ea6-81c2-6c887a3669e2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:55.339798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ed095-30ea-4ea6-81c2-6c887a3669e2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:55.340073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ed095-30ea-4ea6-81c2-6c887a3669e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:55.413878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:55.416834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0a0ed095-30ea-4ea6-81c2-6c887a3669e2 None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:55.418102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0a0ed095-30ea-4ea6-81c2-6c887a3669e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:55.532927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 542/1090] 10.222.0.66 () {58 vars in 1015 bytes} [Fri Oct 28 19:58:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:58.137592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d67360-0dda-4637-90c9-57719cdc2c23 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:58.138172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d67360-0dda-4637-90c9-57719cdc2c23 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:58.138172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d67360-0dda-4637-90c9-57719cdc2c23 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:58.139495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 549/1091] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:58:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:58:58.145610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80d2ab0-fb61-49f6-846c-3167206df362 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:58.146090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80d2ab0-fb61-49f6-846c-3167206df362 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:58.146188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80d2ab0-fb61-49f6-846c-3167206df362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:58.212971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b80d2ab0-fb61-49f6-846c-3167206df362 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:58:58.214225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b80d2ab0-fb61-49f6-846c-3167206df362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:58.269409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:58.311539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 543/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:58:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:58:58.605056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d67360-0dda-4637-90c9-57719cdc2c23 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:58:58.606611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a1d67360-0dda-4637-90c9-57719cdc2c23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:58.754460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d67360-0dda-4637-90c9-57719cdc2c23 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:58:58.827947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:58:58.867926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3974aced-43ce-41f9-95d7-e099b5c0efea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:58:58.868469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3974aced-43ce-41f9-95d7-e099b5c0efea service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:58:58.868750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3974aced-43ce-41f9-95d7-e099b5c0efea service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:58:58.869024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3974aced-43ce-41f9-95d7-e099b5c0efea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:58:58.870635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3974aced-43ce-41f9-95d7-e099b5c0efea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:58:58.958543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3974aced-43ce-41f9-95d7-e099b5c0efea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:58:58.960129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3974aced-43ce-41f9-95d7-e099b5c0efea service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:58:58.978067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 550/1093] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:58:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:00.104222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544bbcda-2074-4640-bdc2-0b9b4f20113a None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:00.104775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544bbcda-2074-4640-bdc2-0b9b4f20113a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:00.104775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544bbcda-2074-4640-bdc2-0b9b4f20113a None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:00.106135 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 544/1094] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:00.110898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4327511-d4de-4a1e-8a31-a97555c6eab8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:00.111357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4327511-d4de-4a1e-8a31-a97555c6eab8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:00.111467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4327511-d4de-4a1e-8a31-a97555c6eab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:00.187200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b4327511-d4de-4a1e-8a31-a97555c6eab8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:00.188401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b4327511-d4de-4a1e-8a31-a97555c6eab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:00.217215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:00.290718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 551/1095] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:59:00.449547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544bbcda-2074-4640-bdc2-0b9b4f20113a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:00.451059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-544bbcda-2074-4640-bdc2-0b9b4f20113a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:00.596123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544bbcda-2074-4640-bdc2-0b9b4f20113a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:00.636607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:00.710899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ec006b-0b8e-4311-a561-dc4aa787a127 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:00.711507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ec006b-0b8e-4311-a561-dc4aa787a127 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:00.711790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ec006b-0b8e-4311-a561-dc4aa787a127 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:00.712065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ec006b-0b8e-4311-a561-dc4aa787a127 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:00.713692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-64ec006b-0b8e-4311-a561-dc4aa787a127 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:00.801829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ec006b-0b8e-4311-a561-dc4aa787a127 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:00.803772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ec006b-0b8e-4311-a561-dc4aa787a127 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:00.822603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 545/1096] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:01.932143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dcb15d-ec75-4ccc-a32f-1029be3a893e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:01.932676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dcb15d-ec75-4ccc-a32f-1029be3a893e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:01.932676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dcb15d-ec75-4ccc-a32f-1029be3a893e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:01.933979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 552/1097] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:01.938836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ea784-387c-46f9-94d7-abcf4ba79a4a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:01.939251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ea784-387c-46f9-94d7-abcf4ba79a4a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:01.939391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ea784-387c-46f9-94d7-abcf4ba79a4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:01.996565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:02.020536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-439ea784-387c-46f9-94d7-abcf4ba79a4a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:02.021800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-439ea784-387c-46f9-94d7-abcf4ba79a4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:02.123128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 546/1098] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:59:02.275946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dcb15d-ec75-4ccc-a32f-1029be3a893e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:02.277560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b9dcb15d-ec75-4ccc-a32f-1029be3a893e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:02.421589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dcb15d-ec75-4ccc-a32f-1029be3a893e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:02.433702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:02.536194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c7e46f-2de7-4e6f-80b7-f6019b20cee0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:02.536721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c7e46f-2de7-4e6f-80b7-f6019b20cee0 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:02.537008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c7e46f-2de7-4e6f-80b7-f6019b20cee0 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:02.537292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c7e46f-2de7-4e6f-80b7-f6019b20cee0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:02.538914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f4c7e46f-2de7-4e6f-80b7-f6019b20cee0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:02.629259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c7e46f-2de7-4e6f-80b7-f6019b20cee0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:02.630898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c7e46f-2de7-4e6f-80b7-f6019b20cee0 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:02.648610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 553/1099] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:03.746552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5cb8be-8320-49e6-bc55-df1edefe8827 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:03.747089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5cb8be-8320-49e6-bc55-df1edefe8827 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:03.747089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5cb8be-8320-49e6-bc55-df1edefe8827 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:03.748454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 547/1100] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:03.753365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be41e8ed-94d6-45da-94ba-1837e1d9c690 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:03.753787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be41e8ed-94d6-45da-94ba-1837e1d9c690 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:03.753908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be41e8ed-94d6-45da-94ba-1837e1d9c690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:03.773010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:03.827519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-be41e8ed-94d6-45da-94ba-1837e1d9c690 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:03.828711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-be41e8ed-94d6-45da-94ba-1837e1d9c690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:03.923594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 554/1101] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:04.083110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5cb8be-8320-49e6-bc55-df1edefe8827 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:04.084615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9c5cb8be-8320-49e6-bc55-df1edefe8827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:04.193935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:04.229862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5cb8be-8320-49e6-bc55-df1edefe8827 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:04.341978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1045cd14-bc14-47de-af85-b62a051f301c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:04.342502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1045cd14-bc14-47de-af85-b62a051f301c service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:04.342781 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1045cd14-bc14-47de-af85-b62a051f301c service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:04.343052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1045cd14-bc14-47de-af85-b62a051f301c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:04.344626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1045cd14-bc14-47de-af85-b62a051f301c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:04.421164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:04.434418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1045cd14-bc14-47de-af85-b62a051f301c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:04.436022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1045cd14-bc14-47de-af85-b62a051f301c service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:04.455291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 548/1102] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:05.559851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92915e29-8d1a-437e-b6b9-158352f18cf8 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:05.560381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92915e29-8d1a-437e-b6b9-158352f18cf8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:05.560381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92915e29-8d1a-437e-b6b9-158352f18cf8 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:05.561711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 555/1103] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:05.566787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73da3285-4686-4c88-919e-a0c9c7968c0c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:05.567196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73da3285-4686-4c88-919e-a0c9c7968c0c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:05.567340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73da3285-4686-4c88-919e-a0c9c7968c0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:05.643615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-73da3285-4686-4c88-919e-a0c9c7968c0c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:05.644858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-73da3285-4686-4c88-919e-a0c9c7968c0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:05.740490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 549/1104] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:59:05.891420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92915e29-8d1a-437e-b6b9-158352f18cf8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:05.892955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-92915e29-8d1a-437e-b6b9-158352f18cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:05.977716 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:06.043827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92915e29-8d1a-437e-b6b9-158352f18cf8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:06.160637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c481708-e0ba-4e9b-a5f3-aacb6ea083d2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:06.161175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c481708-e0ba-4e9b-a5f3-aacb6ea083d2 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:06.161547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c481708-e0ba-4e9b-a5f3-aacb6ea083d2 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:06.161831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c481708-e0ba-4e9b-a5f3-aacb6ea083d2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:06.163516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c481708-e0ba-4e9b-a5f3-aacb6ea083d2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:06.213579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:06.254845 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c481708-e0ba-4e9b-a5f3-aacb6ea083d2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:06.256440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c481708-e0ba-4e9b-a5f3-aacb6ea083d2 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:06.274232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 556/1105] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:07.379310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949d6e7-de31-4e67-942c-f1d1d6d27c85 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:07.379877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949d6e7-de31-4e67-942c-f1d1d6d27c85 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:07.379877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949d6e7-de31-4e67-942c-f1d1d6d27c85 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:07.381225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 550/1106] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:07.386108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a0cdbd-13df-4153-ae2e-7f055425201a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:07.386581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a0cdbd-13df-4153-ae2e-7f055425201a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:07.386686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a0cdbd-13df-4153-ae2e-7f055425201a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:07.464482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-79a0cdbd-13df-4153-ae2e-7f055425201a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:07.465741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-79a0cdbd-13df-4153-ae2e-7f055425201a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:07.562460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 557/1107] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:07.717279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949d6e7-de31-4e67-942c-f1d1d6d27c85 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:07.718832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4949d6e7-de31-4e67-942c-f1d1d6d27c85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:07.767645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:07.866381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949d6e7-de31-4e67-942c-f1d1d6d27c85 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:07.977876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b348cc-1a06-4490-adb6-e2d373e69ab1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:07.978486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b348cc-1a06-4490-adb6-e2d373e69ab1 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:07.978769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b348cc-1a06-4490-adb6-e2d373e69ab1 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:07.979050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b348cc-1a06-4490-adb6-e2d373e69ab1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:07.980649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-87b348cc-1a06-4490-adb6-e2d373e69ab1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:07.988025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:08.072707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b348cc-1a06-4490-adb6-e2d373e69ab1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:08.074325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b348cc-1a06-4490-adb6-e2d373e69ab1 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:08.092235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 551/1108] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:09.216275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e4beaf-613b-483a-b9d5-ec1ebe8a28e5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:09.216842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e4beaf-613b-483a-b9d5-ec1ebe8a28e5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:09.216842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e4beaf-613b-483a-b9d5-ec1ebe8a28e5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:09.218244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 558/1109] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:09.223870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814dc3f4-a954-4e52-b824-dadc55e51b79 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:09.224323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814dc3f4-a954-4e52-b824-dadc55e51b79 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:09.224432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814dc3f4-a954-4e52-b824-dadc55e51b79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:09.301415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-814dc3f4-a954-4e52-b824-dadc55e51b79 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:09.303555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-814dc3f4-a954-4e52-b824-dadc55e51b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:09.371554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:09.400344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 552/1110] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:09.551530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e4beaf-613b-483a-b9d5-ec1ebe8a28e5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:09.553006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-57e4beaf-613b-483a-b9d5-ec1ebe8a28e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:09.571797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:09.701087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e4beaf-613b-483a-b9d5-ec1ebe8a28e5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:09.787349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:09.812848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ab1c92-56d0-4876-b06e-2250cc5bf7ea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:09.813413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab1c92-56d0-4876-b06e-2250cc5bf7ea service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:09.813695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab1c92-56d0-4876-b06e-2250cc5bf7ea service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:09.813970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab1c92-56d0-4876-b06e-2250cc5bf7ea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:09.815946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-89ab1c92-56d0-4876-b06e-2250cc5bf7ea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:09.904619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ab1c92-56d0-4876-b06e-2250cc5bf7ea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:09.906249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ab1c92-56d0-4876-b06e-2250cc5bf7ea service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:09.924539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 559/1111] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:11.026419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209121de-2e0b-415b-b131-65d348f787d2 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:11.026956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209121de-2e0b-415b-b131-65d348f787d2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:11.027071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209121de-2e0b-415b-b131-65d348f787d2 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:11.028432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 553/1112] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:11.033367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0d083-98d6-4bee-be69-4203d59351a3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:11.034444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0d083-98d6-4bee-be69-4203d59351a3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:11.034662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0d083-98d6-4bee-be69-4203d59351a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:11.110363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3cd0d083-98d6-4bee-be69-4203d59351a3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:11.111563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3cd0d083-98d6-4bee-be69-4203d59351a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:11.153300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:11.209248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 560/1113] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:11.360741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209121de-2e0b-415b-b131-65d348f787d2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:11.362429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-209121de-2e0b-415b-b131-65d348f787d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:11.508026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209121de-2e0b-415b-b131-65d348f787d2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:11.562140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:11.620523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864869df-f34f-4b7d-ae3e-2fd265751e5e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:11.621129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864869df-f34f-4b7d-ae3e-2fd265751e5e service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:11.621443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864869df-f34f-4b7d-ae3e-2fd265751e5e service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:11.621718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864869df-f34f-4b7d-ae3e-2fd265751e5e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:11.623298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-864869df-f34f-4b7d-ae3e-2fd265751e5e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:11.713255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864869df-f34f-4b7d-ae3e-2fd265751e5e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:11.714877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864869df-f34f-4b7d-ae3e-2fd265751e5e service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:11.732474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 554/1114] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:12.910577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b1f3a-5e79-4529-9ea3-261d2db1bf83 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:12.911170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b1f3a-5e79-4529-9ea3-261d2db1bf83 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:12.911170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b1f3a-5e79-4529-9ea3-261d2db1bf83 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:12.912512 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 561/1115] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 19:59:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:12.917458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0fc96-5151-4d40-b26a-cc4e48228c7c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:12.917878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0fc96-5151-4d40-b26a-cc4e48228c7c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:12.918012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0fc96-5151-4d40-b26a-cc4e48228c7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:12.992872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5bc0fc96-5151-4d40-b26a-cc4e48228c7c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:12.994087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5bc0fc96-5151-4d40-b26a-cc4e48228c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:12.996057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:13.091825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 555/1116] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 19:59:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:13.240866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b1f3a-5e79-4529-9ea3-261d2db1bf83 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:13.242889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-af1b1f3a-5e79-4529-9ea3-261d2db1bf83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:13.389387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b1f3a-5e79-4529-9ea3-261d2db1bf83 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:13.416299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:13.499320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c151d10-a63e-411f-bf0a-056dde5c0f41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:13.499925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c151d10-a63e-411f-bf0a-056dde5c0f41 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:13.500209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c151d10-a63e-411f-bf0a-056dde5c0f41 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:13.500479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c151d10-a63e-411f-bf0a-056dde5c0f41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:13.502096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2c151d10-a63e-411f-bf0a-056dde5c0f41 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:13.593028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c151d10-a63e-411f-bf0a-056dde5c0f41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:13.594653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c151d10-a63e-411f-bf0a-056dde5c0f41 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:13.613803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 562/1117] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:15.732513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d9940-fc3f-4b51-8831-b0344e5cbc76 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:15.733115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d9940-fc3f-4b51-8831-b0344e5cbc76 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:15.733115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d9940-fc3f-4b51-8831-b0344e5cbc76 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:15.734499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 556/1118] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:15.740296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a487168-43aa-48c0-a79f-85fa587f0526 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:15.740760 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a487168-43aa-48c0-a79f-85fa587f0526 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:15.740873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a487168-43aa-48c0-a79f-85fa587f0526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:15.781482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:15.806376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2a487168-43aa-48c0-a79f-85fa587f0526 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:15.807568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2a487168-43aa-48c0-a79f-85fa587f0526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:15.903188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 563/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:16.198409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d9940-fc3f-4b51-8831-b0344e5cbc76 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:16.200021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f09d9940-fc3f-4b51-8831-b0344e5cbc76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:16.304049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d9940-fc3f-4b51-8831-b0344e5cbc76 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:16.335395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:16.375834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef539d34-724b-4856-ac15-a3a55cfed335 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:16.376485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef539d34-724b-4856-ac15-a3a55cfed335 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:16.377007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef539d34-724b-4856-ac15-a3a55cfed335 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:16.377540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef539d34-724b-4856-ac15-a3a55cfed335 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:16.379592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ef539d34-724b-4856-ac15-a3a55cfed335 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:16.490488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef539d34-724b-4856-ac15-a3a55cfed335 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:16.491913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef539d34-724b-4856-ac15-a3a55cfed335 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:16.509364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 557/1120] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:18.795286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0da23e2-a300-411b-9f78-988c73f0ae97 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:18.795840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0da23e2-a300-411b-9f78-988c73f0ae97 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:18.795840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0da23e2-a300-411b-9f78-988c73f0ae97 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:18.797172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 564/1121] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:18.803107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd35ae-2c64-42a0-9f22-59d07346cac0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:18.803559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd35ae-2c64-42a0-9f22-59d07346cac0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:18.803670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd35ae-2c64-42a0-9f22-59d07346cac0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:18.870415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-fbcd35ae-2c64-42a0-9f22-59d07346cac0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:18.871624 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fbcd35ae-2c64-42a0-9f22-59d07346cac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:18.885509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:18.967934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 558/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:59:19.252640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0da23e2-a300-411b-9f78-988c73f0ae97 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:19.254823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b0da23e2-a300-411b-9f78-988c73f0ae97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:19.359594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0da23e2-a300-411b-9f78-988c73f0ae97 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:19.373187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:19.436364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76b8ec6-62bd-4f38-b278-c0745e64a789 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:19.436968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76b8ec6-62bd-4f38-b278-c0745e64a789 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:19.437250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76b8ec6-62bd-4f38-b278-c0745e64a789 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:19.437557 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76b8ec6-62bd-4f38-b278-c0745e64a789 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:19.439134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e76b8ec6-62bd-4f38-b278-c0745e64a789 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:19.549400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76b8ec6-62bd-4f38-b278-c0745e64a789 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:19.550975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76b8ec6-62bd-4f38-b278-c0745e64a789 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:19.569676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 565/1123] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:22.139333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84de73b-b52b-4f08-96ac-6c1c33d5fb22 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:22.148126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84de73b-b52b-4f08-96ac-6c1c33d5fb22 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:22.148126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84de73b-b52b-4f08-96ac-6c1c33d5fb22 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:22.148126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 559/1124] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:22.148481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f53d6e5-92b5-4d35-a3ca-4a53741384af None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:22.148481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f53d6e5-92b5-4d35-a3ca-4a53741384af None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:22.148481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f53d6e5-92b5-4d35-a3ca-4a53741384af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:22.198468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:22.218282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2f53d6e5-92b5-4d35-a3ca-4a53741384af None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:22.219500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2f53d6e5-92b5-4d35-a3ca-4a53741384af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:22.318142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 566/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:22.607255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84de73b-b52b-4f08-96ac-6c1c33d5fb22 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:22.608772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a84de73b-b52b-4f08-96ac-6c1c33d5fb22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:22.713484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84de73b-b52b-4f08-96ac-6c1c33d5fb22 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:22.752724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:22.786466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2294fc18-a9bd-435a-bb31-b5b94acaf5c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:22.787386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2294fc18-a9bd-435a-bb31-b5b94acaf5c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:22.787876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2294fc18-a9bd-435a-bb31-b5b94acaf5c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:22.788494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2294fc18-a9bd-435a-bb31-b5b94acaf5c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:22.790136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2294fc18-a9bd-435a-bb31-b5b94acaf5c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:22.897913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2294fc18-a9bd-435a-bb31-b5b94acaf5c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:22.899278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2294fc18-a9bd-435a-bb31-b5b94acaf5c1 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:22.917816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 560/1126] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:25.173255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f0c40-23dc-4234-9f6e-154101f672c5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:25.173826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f0c40-23dc-4234-9f6e-154101f672c5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:25.173826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f0c40-23dc-4234-9f6e-154101f672c5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:25.175223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 567/1127] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:25.181126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364b282-c82b-4f24-9ca6-89f1ed8f2b3b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:25.181625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364b282-c82b-4f24-9ca6-89f1ed8f2b3b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:25.181730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2364b282-c82b-4f24-9ca6-89f1ed8f2b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:25.248449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2364b282-c82b-4f24-9ca6-89f1ed8f2b3b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:25.249692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2364b282-c82b-4f24-9ca6-89f1ed8f2b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:25.271372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:25.347645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 561/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:25.632322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999f0c40-23dc-4234-9f6e-154101f672c5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:25.633910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-999f0c40-23dc-4234-9f6e-154101f672c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:25.740076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999f0c40-23dc-4234-9f6e-154101f672c5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:25.760416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:25.813770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a001805-8368-40d8-8839-85269a91a97a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:25.814378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a001805-8368-40d8-8839-85269a91a97a service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:25.814662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a001805-8368-40d8-8839-85269a91a97a service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:25.814937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a001805-8368-40d8-8839-85269a91a97a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:25.816525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3a001805-8368-40d8-8839-85269a91a97a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:25.927106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a001805-8368-40d8-8839-85269a91a97a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:25.928994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a001805-8368-40d8-8839-85269a91a97a service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:25.947477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 568/1129] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:28.236060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719eb611-10aa-4457-8c40-2801ab77c564 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:28.236663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719eb611-10aa-4457-8c40-2801ab77c564 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:28.236663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719eb611-10aa-4457-8c40-2801ab77c564 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:28.238138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 562/1130] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:28.244171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e25c1-3d8f-4e06-bdbf-495847e91b26 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:28.244602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e25c1-3d8f-4e06-bdbf-495847e91b26 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:28.244728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e25c1-3d8f-4e06-bdbf-495847e91b26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:28.314655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:28.317340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-272e25c1-3d8f-4e06-bdbf-495847e91b26 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:28.318579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-272e25c1-3d8f-4e06-bdbf-495847e91b26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:28.415345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 569/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:28.701125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719eb611-10aa-4457-8c40-2801ab77c564 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:28.708757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-719eb611-10aa-4457-8c40-2801ab77c564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:28.804503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719eb611-10aa-4457-8c40-2801ab77c564 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:28.854988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:28.878654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a90f24-982d-4e60-b7eb-a860a949680f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:28.879268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a90f24-982d-4e60-b7eb-a860a949680f service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:28.880091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a90f24-982d-4e60-b7eb-a860a949680f service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:28.880374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a90f24-982d-4e60-b7eb-a860a949680f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:28.882738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-94a90f24-982d-4e60-b7eb-a860a949680f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:28.991280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a90f24-982d-4e60-b7eb-a860a949680f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:28.992675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a90f24-982d-4e60-b7eb-a860a949680f service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:29.011582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 563/1132] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:31.279167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e6c566-da21-465c-9145-5836d33b3928 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:31.279700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e6c566-da21-465c-9145-5836d33b3928 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:31.279700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e6c566-da21-465c-9145-5836d33b3928 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:31.281124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 570/1133] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:31.286492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81664a06-5c97-4a4b-82be-0b14f395fe74 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:31.287535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81664a06-5c97-4a4b-82be-0b14f395fe74 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:31.287535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81664a06-5c97-4a4b-82be-0b14f395fe74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:31.355630 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-81664a06-5c97-4a4b-82be-0b14f395fe74 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:31.356843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-81664a06-5c97-4a4b-82be-0b14f395fe74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:31.386202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:31.453701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 564/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:31.736857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e6c566-da21-465c-9145-5836d33b3928 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:31.738396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-47e6c566-da21-465c-9145-5836d33b3928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:31.841568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e6c566-da21-465c-9145-5836d33b3928 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:31.869623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:31.915921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324d2316-eba6-4e2c-8fda-1f807083fa9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:31.916527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324d2316-eba6-4e2c-8fda-1f807083fa9f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:31.916812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324d2316-eba6-4e2c-8fda-1f807083fa9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:31.917086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324d2316-eba6-4e2c-8fda-1f807083fa9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:31.918698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-324d2316-eba6-4e2c-8fda-1f807083fa9f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:32.027679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324d2316-eba6-4e2c-8fda-1f807083fa9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:32.029115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324d2316-eba6-4e2c-8fda-1f807083fa9f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:32.046963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 571/1135] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:34.337395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda89fd3-a0b2-4176-a675-3aafc50fc186 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:34.337923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda89fd3-a0b2-4176-a675-3aafc50fc186 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:34.337923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda89fd3-a0b2-4176-a675-3aafc50fc186 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:34.339254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 565/1136] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:34.344604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0847a98-6142-48e7-9137-fa0082cc8a73 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:34.345027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0847a98-6142-48e7-9137-fa0082cc8a73 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:34.345166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0847a98-6142-48e7-9137-fa0082cc8a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:34.411166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f0847a98-6142-48e7-9137-fa0082cc8a73 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:34.412978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f0847a98-6142-48e7-9137-fa0082cc8a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:34.421212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:34.511697 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 572/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:34.796780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda89fd3-a0b2-4176-a675-3aafc50fc186 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:34.798374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-cda89fd3-a0b2-4176-a675-3aafc50fc186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:34.902542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda89fd3-a0b2-4176-a675-3aafc50fc186 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:34.958181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:34.976172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50796f-c79d-4b87-9993-94045210c558 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:34.978258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50796f-c79d-4b87-9993-94045210c558 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:34.978258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50796f-c79d-4b87-9993-94045210c558 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:34.978258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50796f-c79d-4b87-9993-94045210c558 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:34.978888 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2e50796f-c79d-4b87-9993-94045210c558 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:35.089692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e50796f-c79d-4b87-9993-94045210c558 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:35.091064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e50796f-c79d-4b87-9993-94045210c558 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:35.108918 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 566/1138] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:37.301828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008b1894-6267-45b5-988d-3192aa8cfc4f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:37.302946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008b1894-6267-45b5-988d-3192aa8cfc4f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:37.302946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008b1894-6267-45b5-988d-3192aa8cfc4f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:37.304318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 573/1139] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 19:59:37.309622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f85b8d-eb84-4156-bbe4-044a863c769d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:37.310057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f85b8d-eb84-4156-bbe4-044a863c769d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:37.310193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f85b8d-eb84-4156-bbe4-044a863c769d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:37.375938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-49f85b8d-eb84-4156-bbe4-044a863c769d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:37.377144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-49f85b8d-eb84-4156-bbe4-044a863c769d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:37.415365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:37.474356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 567/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 19:59:37.759742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008b1894-6267-45b5-988d-3192aa8cfc4f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:37.761265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-008b1894-6267-45b5-988d-3192aa8cfc4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:37.862238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008b1894-6267-45b5-988d-3192aa8cfc4f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:37.899392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:37.933296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364ac8a-6a68-4d37-be0c-a7b0045c6139 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:37.933930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364ac8a-6a68-4d37-be0c-a7b0045c6139 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:37.934209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364ac8a-6a68-4d37-be0c-a7b0045c6139 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:37.934483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364ac8a-6a68-4d37-be0c-a7b0045c6139 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:37.936050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6364ac8a-6a68-4d37-be0c-a7b0045c6139 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:38.044706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364ac8a-6a68-4d37-be0c-a7b0045c6139 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:38.046179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364ac8a-6a68-4d37-be0c-a7b0045c6139 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:38.064594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 574/1141] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:40.230655 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf77a9d-f9de-413b-a2e7-75e6704edb2e None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:40.231243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf77a9d-f9de-413b-a2e7-75e6704edb2e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:40.231243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf77a9d-f9de-413b-a2e7-75e6704edb2e None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:40.232567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 568/1142] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:40.238038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adca5bef-ed64-4f04-9fd8-1aa8d53fbb0d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:40.238440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adca5bef-ed64-4f04-9fd8-1aa8d53fbb0d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:40.238623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adca5bef-ed64-4f04-9fd8-1aa8d53fbb0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:40.304548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-adca5bef-ed64-4f04-9fd8-1aa8d53fbb0d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:40.305695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-adca5bef-ed64-4f04-9fd8-1aa8d53fbb0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:40.324470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:40.409810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 575/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:40.692631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf77a9d-f9de-413b-a2e7-75e6704edb2e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:40.694989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7bf77a9d-f9de-413b-a2e7-75e6704edb2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:40.798634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf77a9d-f9de-413b-a2e7-75e6704edb2e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:40.866154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:40.872933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adcc0f6-da83-4bde-be77-f0fd8fab142f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:40.873570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adcc0f6-da83-4bde-be77-f0fd8fab142f service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:40.873852 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adcc0f6-da83-4bde-be77-f0fd8fab142f service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:40.874149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adcc0f6-da83-4bde-be77-f0fd8fab142f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:40.876174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7adcc0f6-da83-4bde-be77-f0fd8fab142f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:40.987175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adcc0f6-da83-4bde-be77-f0fd8fab142f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:40.988633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adcc0f6-da83-4bde-be77-f0fd8fab142f service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:41.006732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 569/1144] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:43.148731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db17351f-bdc6-4450-aeb0-6e54b03d3c9d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:43.149289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db17351f-bdc6-4450-aeb0-6e54b03d3c9d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:43.149289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db17351f-bdc6-4450-aeb0-6e54b03d3c9d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:43.150681 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 576/1145] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 19:59:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 19:59:43.155957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c34bd3f-6b44-4c44-a19e-1e712f7d72fc None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:43.156391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c34bd3f-6b44-4c44-a19e-1e712f7d72fc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:43.156512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c34bd3f-6b44-4c44-a19e-1e712f7d72fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:43.223752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8c34bd3f-6b44-4c44-a19e-1e712f7d72fc None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 19:59:43.225671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8c34bd3f-6b44-4c44-a19e-1e712f7d72fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:43.273968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:43.323223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 570/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 19:59:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 19:59:43.607068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db17351f-bdc6-4450-aeb0-6e54b03d3c9d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 19:59:43.608611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-db17351f-bdc6-4450-aeb0-6e54b03d3c9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:43.712480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db17351f-bdc6-4450-aeb0-6e54b03d3c9d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 19:59:43.759821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:43.787071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6128a6cc-a791-4241-96b1-8f28f0b73afa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 19:59:43.787800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128a6cc-a791-4241-96b1-8f28f0b73afa service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 19:59:43.788094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128a6cc-a791-4241-96b1-8f28f0b73afa service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 19:59:43.788623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128a6cc-a791-4241-96b1-8f28f0b73afa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 19:59:43.790788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6128a6cc-a791-4241-96b1-8f28f0b73afa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 19:59:43.898532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6128a6cc-a791-4241-96b1-8f28f0b73afa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 19:59:43.899875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6128a6cc-a791-4241-96b1-8f28f0b73afa service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 19:59:43.917687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 577/1147] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 19:59:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 19:59:53.321621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=22 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 19:59:53.915448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=66 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:04.894306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e8b8be-8583-429f-9667-552aa1fc6303 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:04.894849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e8b8be-8583-429f-9667-552aa1fc6303 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:04.894849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e8b8be-8583-429f-9667-552aa1fc6303 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:04.897053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 571/1148] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 20:00:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:04.902409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77b4d0-c5ac-4328-bf44-4b715df89674 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:04.902844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77b4d0-c5ac-4328-bf44-4b715df89674 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:04.902958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77b4d0-c5ac-4328-bf44-4b715df89674 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:04.967738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bf77b4d0-c5ac-4328-bf44-4b715df89674 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:04.968918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bf77b4d0-c5ac-4328-bf44-4b715df89674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:05.066204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 578/1149] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 20:00:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:00:05.372873 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10231f5b-dd9d-44c6-934e-bc69a3c620be None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:05.373161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10231f5b-dd9d-44c6-934e-bc69a3c620be None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:05.373480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10231f5b-dd9d-44c6-934e-bc69a3c620be None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:05.374833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 572/1150] 10.222.0.66 () {54 vars in 962 bytes} [Fri Oct 28 20:00:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:05.379712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f643d2-ae53-422e-b166-483c922fe37c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:05.380667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f643d2-ae53-422e-b166-483c922fe37c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:05.380980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f643d2-ae53-422e-b166-483c922fe37c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:05.456484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-58f643d2-ae53-422e-b166-483c922fe37c None None] MFA Rules not processed for user `a7fae443ad92439aadd2f94ad3bca594`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:05.457836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-58f643d2-ae53-422e-b166-483c922fe37c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:05.478671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:05.533153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 579/1151] 10.222.0.66 () {58 vars in 1059 bytes} [Fri Oct 28 20:00:05 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:00:05.541650 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10231f5b-dd9d-44c6-934e-bc69a3c620be None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:00:05.543154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-10231f5b-dd9d-44c6-934e-bc69a3c620be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:05.647334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10231f5b-dd9d-44c6-934e-bc69a3c620be None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:00:05.721903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894b923a-7fb0-4492-b807-20421b667536 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7fae443ad92439aadd2f94ad3bca594', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:00:05.722509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b923a-7fb0-4492-b807-20421b667536 service glance] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:05.722790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b923a-7fb0-4492-b807-20421b667536 service glance] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:05.723063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894b923a-7fb0-4492-b807-20421b667536 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:05.725369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-894b923a-7fb0-4492-b807-20421b667536 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:05.779485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:05.835860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894b923a-7fb0-4492-b807-20421b667536 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:00:05.837307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894b923a-7fb0-4492-b807-20421b667536 service glance] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:00:05.855982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 573/1152] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:00:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:00:08.100253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1604207b-bcfd-4ba0-8cbb-c1f16fe89495 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:08.100792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1604207b-bcfd-4ba0-8cbb-c1f16fe89495 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:08.100792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1604207b-bcfd-4ba0-8cbb-c1f16fe89495 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:08.102255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 580/1153] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 20:00:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:08.107665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdfb4e7-37c1-44cc-8a5e-5735c7f35ec1 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:08.108107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdfb4e7-37c1-44cc-8a5e-5735c7f35ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:08.108205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdfb4e7-37c1-44cc-8a5e-5735c7f35ec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:08.174154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ecdfb4e7-37c1-44cc-8a5e-5735c7f35ec1 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:08.175345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ecdfb4e7-37c1-44cc-8a5e-5735c7f35ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:08.271978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 574/1154] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 20:00:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:00:08.542961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1604207b-bcfd-4ba0-8cbb-c1f16fe89495 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:00:08.544496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1604207b-bcfd-4ba0-8cbb-c1f16fe89495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:08.670644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1604207b-bcfd-4ba0-8cbb-c1f16fe89495 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:00:08.720375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:08.768993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f832e-0ec5-4b2c-8780-f8d69eb66bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:00:08.769664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f832e-0ec5-4b2c-8780-f8d69eb66bb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:08.769947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f832e-0ec5-4b2c-8780-f8d69eb66bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:08.770221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f832e-0ec5-4b2c-8780-f8d69eb66bb5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:08.770861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f832e-0ec5-4b2c-8780-f8d69eb66bb5 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:00:08.772307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-c17f832e-0ec5-4b2c-8780-f8d69eb66bb5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:00:08.772307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 20:00:08.772307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 20:00:08.773601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f832e-0ec5-4b2c-8780-f8d69eb66bb5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:00:08.793043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 581/1155] 10.222.0.66 () {56 vars in 1108 bytes} [Fri Oct 28 20:00:08 2022] GET /identity/v3/projects => generated 2553 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:00:10.975501 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236cd52b-2160-482b-9dad-d10719f445c6 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:10.976077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236cd52b-2160-482b-9dad-d10719f445c6 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:10.976077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236cd52b-2160-482b-9dad-d10719f445c6 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:10.977428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 575/1156] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 20:00:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:10.982608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b7f52-de17-4cca-a0d5-66d598b51083 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:10.983028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b7f52-de17-4cca-a0d5-66d598b51083 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:10.983170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b7f52-de17-4cca-a0d5-66d598b51083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:11.049973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-069b7f52-de17-4cca-a0d5-66d598b51083 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:11.051189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-069b7f52-de17-4cca-a0d5-66d598b51083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:11.149059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 582/1157] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 20:00:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:00:11.446959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236cd52b-2160-482b-9dad-d10719f445c6 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:00:11.448462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-236cd52b-2160-482b-9dad-d10719f445c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:11.458766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:11.594965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236cd52b-2160-482b-9dad-d10719f445c6 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:00:11.675773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:11.710132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37c005c-6639-4df1-ac29-7ee3efeb8512 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:00:11.710739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37c005c-6639-4df1-ac29-7ee3efeb8512 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:11.711020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37c005c-6639-4df1-ac29-7ee3efeb8512 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:11.711294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37c005c-6639-4df1-ac29-7ee3efeb8512 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:11.712882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b37c005c-6639-4df1-ac29-7ee3efeb8512 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:11.803946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37c005c-6639-4df1-ac29-7ee3efeb8512 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:00:11.805666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37c005c-6639-4df1-ac29-7ee3efeb8512 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:00:11.823930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 576/1158] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:00:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:00:14.160390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8519318-fa57-4f0f-853c-0694140cba25 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:14.160973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8519318-fa57-4f0f-853c-0694140cba25 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:14.161132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8519318-fa57-4f0f-853c-0694140cba25 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:14.162470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 583/1159] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 20:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:14.167708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf53a88-4382-491f-9b67-c9cae4fdab1f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:14.168153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf53a88-4382-491f-9b67-c9cae4fdab1f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:14.168244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf53a88-4382-491f-9b67-c9cae4fdab1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:14.233842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-dbf53a88-4382-491f-9b67-c9cae4fdab1f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:14.235061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-dbf53a88-4382-491f-9b67-c9cae4fdab1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:14.281215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:14.331773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 577/1160] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 20:00:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:00:14.630063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8519318-fa57-4f0f-853c-0694140cba25 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:00:14.631777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d8519318-fa57-4f0f-853c-0694140cba25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:14.655033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:14.734448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8519318-fa57-4f0f-853c-0694140cba25 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:00:14.807614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fd7eaf-718c-4b6a-90d3-54b4bb8e58ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:00:14.808223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd7eaf-718c-4b6a-90d3-54b4bb8e58ed service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:14.808505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd7eaf-718c-4b6a-90d3-54b4bb8e58ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:14.808776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd7eaf-718c-4b6a-90d3-54b4bb8e58ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:14.810391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b1fd7eaf-718c-4b6a-90d3-54b4bb8e58ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:14.883375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:14.917760 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fd7eaf-718c-4b6a-90d3-54b4bb8e58ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:00:14.919190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fd7eaf-718c-4b6a-90d3-54b4bb8e58ed service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:00:14.936729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 584/1161] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:00:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:00:17.156185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e334454f-b088-43c3-9660-1209ed975696 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:17.156849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e334454f-b088-43c3-9660-1209ed975696 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:17.156849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e334454f-b088-43c3-9660-1209ed975696 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:17.158125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 578/1162] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 20:00:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:17.163990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7efcce0-2509-4454-b5bb-e017a208c8a3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:17.164441 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7efcce0-2509-4454-b5bb-e017a208c8a3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:17.164549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7efcce0-2509-4454-b5bb-e017a208c8a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:17.231908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b7efcce0-2509-4454-b5bb-e017a208c8a3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:17.233114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b7efcce0-2509-4454-b5bb-e017a208c8a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:17.328304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 585/1163] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 20:00:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:00:17.618611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e334454f-b088-43c3-9660-1209ed975696 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:00:17.620172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e334454f-b088-43c3-9660-1209ed975696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:17.724952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e334454f-b088-43c3-9660-1209ed975696 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:00:17.796987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:17.798462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281179af-084e-4181-9b53-7bd9484ce41e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:00:17.799102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281179af-084e-4181-9b53-7bd9484ce41e service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:17.799385 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281179af-084e-4181-9b53-7bd9484ce41e service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:17.799663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281179af-084e-4181-9b53-7bd9484ce41e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:17.801242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-281179af-084e-4181-9b53-7bd9484ce41e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:17.912770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281179af-084e-4181-9b53-7bd9484ce41e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:00:17.914161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281179af-084e-4181-9b53-7bd9484ce41e service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:00:17.932426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 579/1164] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:00:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:00:20.262071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8addc01d-933f-4a59-aab6-9924f60c4312 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:20.262603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8addc01d-933f-4a59-aab6-9924f60c4312 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:20.262603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8addc01d-933f-4a59-aab6-9924f60c4312 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:20.264018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 586/1165] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 20:00:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:20.269453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e4d042-0053-43b0-95d9-88caaa542bac None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:20.269894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e4d042-0053-43b0-95d9-88caaa542bac None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:20.270009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e4d042-0053-43b0-95d9-88caaa542bac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:20.336591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d2e4d042-0053-43b0-95d9-88caaa542bac None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:20.337750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d2e4d042-0053-43b0-95d9-88caaa542bac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:20.388703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:20.433446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 580/1166] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 20:00:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:00:20.735982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8addc01d-933f-4a59-aab6-9924f60c4312 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:00:20.737565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8addc01d-933f-4a59-aab6-9924f60c4312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:20.774464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:20.841440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8addc01d-933f-4a59-aab6-9924f60c4312 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:00:20.915850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c247ff8f-0b5a-4daa-bcfb-2dd28c862c55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:00:20.916466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247ff8f-0b5a-4daa-bcfb-2dd28c862c55 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:20.916750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247ff8f-0b5a-4daa-bcfb-2dd28c862c55 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:20.917031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247ff8f-0b5a-4daa-bcfb-2dd28c862c55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:20.918687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c247ff8f-0b5a-4daa-bcfb-2dd28c862c55 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:20.998442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:21.029109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c247ff8f-0b5a-4daa-bcfb-2dd28c862c55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:00:21.030631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c247ff8f-0b5a-4daa-bcfb-2dd28c862c55 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:00:21.050008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 587/1167] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:00:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:00:23.661501 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71e4d77-5808-4316-9452-d8552e377376 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:23.662036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71e4d77-5808-4316-9452-d8552e377376 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:23.662036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71e4d77-5808-4316-9452-d8552e377376 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:23.663298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 581/1168] 10.222.0.66 () {54 vars in 927 bytes} [Fri Oct 28 20:00:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:00:23.668503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e59997b-f3ac-4f8b-b75d-3eff8e39e3d4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:23.668923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e59997b-f3ac-4f8b-b75d-3eff8e39e3d4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:23.669059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e59997b-f3ac-4f8b-b75d-3eff8e39e3d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:23.733802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0e59997b-f3ac-4f8b-b75d-3eff8e39e3d4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:00:23.734992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0e59997b-f3ac-4f8b-b75d-3eff8e39e3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:23.830650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 588/1169] 10.222.0.66 () {58 vars in 1024 bytes} [Fri Oct 28 20:00:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:00:24.118038 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71e4d77-5808-4316-9452-d8552e377376 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:00:24.119567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b71e4d77-5808-4316-9452-d8552e377376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:24.222886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71e4d77-5808-4316-9452-d8552e377376 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:00:24.296840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20526d09-531d-484f-993a-45644da6bece service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:00:24.297405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20526d09-531d-484f-993a-45644da6bece service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:00:24.297694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20526d09-531d-484f-993a-45644da6bece service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:00:24.297969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20526d09-531d-484f-993a-45644da6bece service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:00:24.299566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-20526d09-531d-484f-993a-45644da6bece service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:00:24.308329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:24.411167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20526d09-531d-484f-993a-45644da6bece service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:00:24.412687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20526d09-531d-484f-993a-45644da6bece service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:00:24.430830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 582/1170] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:00:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:00:33.829072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=82 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:00:34.428239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=50 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:52.759067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:52.760262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:52.760262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:52.836473 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:01:52.837728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:52.936903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 589/1171] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:01:52.944890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a17d70-46c7-4506-a3e7-9336f3cead64 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:52.945396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a17d70-46c7-4506-a3e7-9336f3cead64 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:52.945505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a17d70-46c7-4506-a3e7-9336f3cead64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:53.021637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-76a17d70-46c7-4506-a3e7-9336f3cead64 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:01:53.022757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-76a17d70-46c7-4506-a3e7-9336f3cead64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:53.120166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 583/1172] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:01:53.125004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:53.126251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:53.138554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdcdf84-1b6d-4a68-9527-5e8bb33a688d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:53.201828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:53.235314 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:53.235929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:53.236210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:53.236485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:53.237127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:01:53.237458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:53.238471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:01:53.238471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 20:01:53.238471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 20:01:53.239348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:53.250457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 590/1173] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:01:53.260826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a17d70-46c7-4506-a3e7-9336f3cead64 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:53.270136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a17d70-46c7-4506-a3e7-9336f3cead64 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:53.332766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:53.365285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:53.365948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:53.366223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:53.366487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:53.367097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:53.368196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:01:53.368196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 20:01:53.368196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 20:01:53.369425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:53.487023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 584/1174] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:01:53 2022] POST /identity/v3/projects => generated 379 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:01:53.491422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:53.492944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:53.527386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4cf67-3482-4d40-8769-946e788f0761 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:53.625750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:53.626279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:53.626561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:53.626833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:53.627506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:53.628518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:01:53.628518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 20:01:53.628518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 20:01:53.629767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:53.784433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 591/1175] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:01:53 2022] POST /identity/v3/users => generated 290 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:01:53.788529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:53.790065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:53.851612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421a813-f6c9-4553-908e-cdb7969da8c1 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:53.934860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:53.944493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:53.949454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:53.949454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:53.949454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:53.949454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:53.949454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:53.961202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 585/1176] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:01:53.965476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:53.967010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:53.979056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b121ad0-a445-41a1-a20d-336d15c66928 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:54.004148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:54.074808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb383b2-3a02-4f96-9c87-e422ea5e0319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:54.075422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb383b2-3a02-4f96-9c87-e422ea5e0319 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:54.075697 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb383b2-3a02-4f96-9c87-e422ea5e0319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:54.075966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb383b2-3a02-4f96-9c87-e422ea5e0319 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:54.076601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb383b2-3a02-4f96-9c87-e422ea5e0319 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:54.078218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb383b2-3a02-4f96-9c87-e422ea5e0319 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:54.090308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 592/1177] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:01:54.094541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:54.096067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:54.129815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc28c141-e2b5-4c09-990c-c7acd8abc0b4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:54.223121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfda2df3-57ca-45b7-9320-f4d7f9bca584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:54.223777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfda2df3-57ca-45b7-9320-f4d7f9bca584 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:54.224052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfda2df3-57ca-45b7-9320-f4d7f9bca584 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:54.224317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfda2df3-57ca-45b7-9320-f4d7f9bca584 admin admin] PATH_INFO: `/v3/projects/9f70234f6f31411dbed3157146698891/users/c2ad521915794ee88378ec76b5ae17fe/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:54.247436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfda2df3-57ca-45b7-9320-f4d7f9bca584 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f70234f6f31411dbed3157146698891, user_id=c2ad521915794ee88378ec76b5ae17fe, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:54.250110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfda2df3-57ca-45b7-9320-f4d7f9bca584 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:54.356615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 586/1178] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:01:54 2022] PUT /identity/v3/projects/9f70234f6f31411dbed3157146698891/users/c2ad521915794ee88378ec76b5ae17fe/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:01:54.367723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c62eb9-bf15-4247-878f-f87fac59d943 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:54.368015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c62eb9-bf15-4247-878f-f87fac59d943 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:54.368293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c62eb9-bf15-4247-878f-f87fac59d943 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:54.425440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-06c62eb9-bf15-4247-878f-f87fac59d943 None None] MFA Rules not processed for user `c2ad521915794ee88378ec76b5ae17fe`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:01:54.426654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-06c62eb9-bf15-4247-878f-f87fac59d943 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:54.504033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 593/1179] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:01:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:01:54.517783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a37e38-e124-4697-9aa0-07287e8acf88 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:54.518084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a37e38-e124-4697-9aa0-07287e8acf88 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:54.518361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a37e38-e124-4697-9aa0-07287e8acf88 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:54.519728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 587/1180] 10.222.0.66 () {54 vars in 963 bytes} [Fri Oct 28 20:01:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:01:54.524655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcc3ca3-4c8a-482b-b64e-65d126528cf1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:54.524947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcc3ca3-4c8a-482b-b64e-65d126528cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:54.525288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcc3ca3-4c8a-482b-b64e-65d126528cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:54.558105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:54.602154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1fcc3ca3-4c8a-482b-b64e-65d126528cf1 None None] MFA Rules not processed for user `183a4f23782143b0b234d7800b409452`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:01:54.604048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1fcc3ca3-4c8a-482b-b64e-65d126528cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:54.680789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 594/1181] 10.222.0.66 () {58 vars in 1060 bytes} [Fri Oct 28 20:01:54 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:01:54.688698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a37e38-e124-4697-9aa0-07287e8acf88 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:54.690238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-80a37e38-e124-4697-9aa0-07287e8acf88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:54.742000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:54.795707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a37e38-e124-4697-9aa0-07287e8acf88 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:54.869064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:54.869695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:54.869975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:54.870243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:54.871814 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:54.968444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:54.969904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:54.973276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:54.987087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 588/1182] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:01:54 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:01:55.033309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a36737-d8f0-41b6-8771-e89c67648549 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:55.033765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a36737-d8f0-41b6-8771-e89c67648549 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:55.033906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a36737-d8f0-41b6-8771-e89c67648549 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:55.035168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 595/1183] 10.222.0.66 () {52 vars in 830 bytes} [Fri Oct 28 20:01:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:01:55.038688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:55.039892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:55.139248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdb25bc-4a63-4a47-82b2-93891bb4e3c5 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:55.235198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:55.235824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:55.236112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:55.236387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] PATH_INFO: `/v3/users/c2ad521915794ee88378ec76b5ae17fe` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:55.250084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2ad521915794ee88378ec76b5ae17fe)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:55.251149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:01:55.251149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 20:01:55.251149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 20:01:55.252369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:55.263473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:55.433758 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2ad521915794ee88378ec76b5ae17fe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:55.434132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2ad521915794ee88378ec76b5ae17fe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:55.457570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2ad521915794ee88378ec76b5ae17fe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:55.471779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2ad521915794ee88378ec76b5ae17fe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:55.472346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2ad521915794ee88378ec76b5ae17fe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:55.543380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 589/1184] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:01:55 2022] DELETE /identity/v3/users/c2ad521915794ee88378ec76b5ae17fe => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:01:55.547799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:55.550892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:55.550892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:55.607352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:01:55.608571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:55.685959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:55.698419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 596/1185] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:01:55.710687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:55.711930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:55.814068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e397d-a5b8-4204-9cc5-3dccfb85b6d0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:55.867724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:01:55.887708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96241c01-efe0-4f2d-aa5f-42a9423bc24e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:55.888348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96241c01-efe0-4f2d-aa5f-42a9423bc24e service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:55.888633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96241c01-efe0-4f2d-aa5f-42a9423bc24e service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:55.888906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96241c01-efe0-4f2d-aa5f-42a9423bc24e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:55.890587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-96241c01-efe0-4f2d-aa5f-42a9423bc24e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:55.998369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96241c01-efe0-4f2d-aa5f-42a9423bc24e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:55.999798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96241c01-efe0-4f2d-aa5f-42a9423bc24e service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:56.019237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 590/1186] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:01:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:01:56.472371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:01:56.472865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:01:56.485085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9cb396-4756-425f-a4b5-d0a3ee07e271 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:01:56.580362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:01:56.580916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:01:56.581194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:01:56.581502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] PATH_INFO: `/v3/projects/9f70234f6f31411dbed3157146698891` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:01:56.591759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f70234f6f31411dbed3157146698891)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:01:56.592779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:01:56.592779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 20:01:56.592779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 20:01:56.594019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:01:56.703448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f70234f6f31411dbed3157146698891', 'request_id': 'req-9542926c-3b39-4877-aae8-3d6f07deee06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:56.764660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f70234f6f31411dbed3157146698891', 'request_id': 'req-9542926c-3b39-4877-aae8-3d6f07deee06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:56.779481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f70234f6f31411dbed3157146698891', 'request_id': 'req-9542926c-3b39-4877-aae8-3d6f07deee06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:56.802814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] The token cache is being invalidate because project 9f70234f6f31411dbed3157146698891 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:01:56.803094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:56.803524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9542926c-3b39-4877-aae8-3d6f07deee06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:01:56.804987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 597/1187] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:01:56 2022] DELETE /identity/v3/projects/9f70234f6f31411dbed3157146698891 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:02:06.016827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=62 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:02:06.800829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:03:24.080945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9382d6a-0f63-4e92-b2b2-4ff8bcc41526 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:24.081511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9382d6a-0f63-4e92-b2b2-4ff8bcc41526 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:24.081511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9382d6a-0f63-4e92-b2b2-4ff8bcc41526 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:24.082869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 591/1188] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 20:03:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:24.088165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa7d35-d121-40b3-bbdc-d3a608ae636b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:24.088584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa7d35-d121-40b3-bbdc-d3a608ae636b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:24.088846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa7d35-d121-40b3-bbdc-d3a608ae636b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:24.169259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-52fa7d35-d121-40b3-bbdc-d3a608ae636b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:24.170508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-52fa7d35-d121-40b3-bbdc-d3a608ae636b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:24.273961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 598/1189] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 20:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:24.428493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9382d6a-0f63-4e92-b2b2-4ff8bcc41526 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:24.430672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a9382d6a-0f63-4e92-b2b2-4ff8bcc41526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:24.552776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9382d6a-0f63-4e92-b2b2-4ff8bcc41526 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:24.558459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42df0aac-5e44-43ee-8ca9-b215ad4c3b15 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:24.559071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42df0aac-5e44-43ee-8ca9-b215ad4c3b15 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:24.559408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42df0aac-5e44-43ee-8ca9-b215ad4c3b15 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:24.559683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42df0aac-5e44-43ee-8ca9-b215ad4c3b15 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:24.568247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42df0aac-5e44-43ee-8ca9-b215ad4c3b15 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:24.572984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42df0aac-5e44-43ee-8ca9-b215ad4c3b15 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:24.580347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 592/1190] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:25.695789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b94c86-b622-434d-a931-eea0cca65493 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:25.696342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b94c86-b622-434d-a931-eea0cca65493 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:25.696342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b94c86-b622-434d-a931-eea0cca65493 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:25.698464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 599/1191] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 20:03:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:25.703641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ce265-e7b6-4352-acc5-eb1c0ea74ae4 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:25.707375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ce265-e7b6-4352-acc5-eb1c0ea74ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:25.707375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ce265-e7b6-4352-acc5-eb1c0ea74ae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:25.725548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-638ce265-e7b6-4352-acc5-eb1c0ea74ae4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:25.726801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-638ce265-e7b6-4352-acc5-eb1c0ea74ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:25.736618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 593/1192] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 20:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:25.895490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b94c86-b622-434d-a931-eea0cca65493 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:25.909649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b94c86-b622-434d-a931-eea0cca65493 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:25.914902 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88d81a3-bdeb-48d7-8ff8-c00dd81c039c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:25.915498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d81a3-bdeb-48d7-8ff8-c00dd81c039c service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:25.915801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d81a3-bdeb-48d7-8ff8-c00dd81c039c service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:25.916072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88d81a3-bdeb-48d7-8ff8-c00dd81c039c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:25.924909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88d81a3-bdeb-48d7-8ff8-c00dd81c039c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:25.926775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88d81a3-bdeb-48d7-8ff8-c00dd81c039c service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:25.938175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 600/1193] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:26.019610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96a2a30-2de3-4ca5-9225-e7494b30991e None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:26.019930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96a2a30-2de3-4ca5-9225-e7494b30991e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:26.020205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96a2a30-2de3-4ca5-9225-e7494b30991e None None] PATH_INFO: `/v3` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:26.021542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 594/1194] 10.222.0.62 () {54 vars in 925 bytes} [Fri Oct 28 20:03:26 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:03:26.028802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be702e0f-86b6-4c31-a64e-cec9420c828a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:26.029124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be702e0f-86b6-4c31-a64e-cec9420c828a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:26.029400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be702e0f-86b6-4c31-a64e-cec9420c828a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:26.091844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-be702e0f-86b6-4c31-a64e-cec9420c828a None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:26.093168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-be702e0f-86b6-4c31-a64e-cec9420c828a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:26.109970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:03:26.163926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 601/1195] 10.222.0.62 () {58 vars in 1015 bytes} [Fri Oct 28 20:03:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:29.946369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49757441-24f6-4edf-8c33-4d6e84594789 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:29.947346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49757441-24f6-4edf-8c33-4d6e84594789 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:29.947480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49757441-24f6-4edf-8c33-4d6e84594789 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:29.948848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 595/1196] 10.222.0.66 () {54 vars in 961 bytes} [Fri Oct 28 20:03:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:29.954292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25e6842-3d1e-47d1-aecb-4355d9db8679 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:29.954728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25e6842-3d1e-47d1-aecb-4355d9db8679 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:29.954855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25e6842-3d1e-47d1-aecb-4355d9db8679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:29.974865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b25e6842-3d1e-47d1-aecb-4355d9db8679 None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:29.976081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b25e6842-3d1e-47d1-aecb-4355d9db8679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:29.985168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 602/1197] 10.222.0.66 () {58 vars in 1058 bytes} [Fri Oct 28 20:03:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:29.992646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49757441-24f6-4edf-8c33-4d6e84594789 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:30.005914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49757441-24f6-4edf-8c33-4d6e84594789 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:30.013223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162d6305-13de-41d5-a301-954ea51cba7a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:30.013859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d6305-13de-41d5-a301-954ea51cba7a service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:30.014143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d6305-13de-41d5-a301-954ea51cba7a service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:30.014416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d6305-13de-41d5-a301-954ea51cba7a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:30.022835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162d6305-13de-41d5-a301-954ea51cba7a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:30.024410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162d6305-13de-41d5-a301-954ea51cba7a service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:30.035014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 596/1198] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:37.513391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f00a55-486d-4af7-8661-20439cc0d69f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:37.514002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f00a55-486d-4af7-8661-20439cc0d69f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:37.514002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f00a55-486d-4af7-8661-20439cc0d69f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:37.515299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 603/1199] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 20:03:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:37.520403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6479f43-76fe-4021-8206-cadc9c8a297b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:37.520828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6479f43-76fe-4021-8206-cadc9c8a297b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:37.520937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6479f43-76fe-4021-8206-cadc9c8a297b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:37.540685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e6479f43-76fe-4021-8206-cadc9c8a297b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:37.541889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e6479f43-76fe-4021-8206-cadc9c8a297b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:37.549925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 597/1200] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 20:03:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:37.703875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f00a55-486d-4af7-8661-20439cc0d69f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:37.720341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f00a55-486d-4af7-8661-20439cc0d69f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:37.736037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3b864-ff02-4fd7-b531-c4e7d8a53463 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:37.736653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3b864-ff02-4fd7-b531-c4e7d8a53463 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:37.736941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3b864-ff02-4fd7-b531-c4e7d8a53463 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:37.737222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3b864-ff02-4fd7-b531-c4e7d8a53463 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:37.745914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d3b864-ff02-4fd7-b531-c4e7d8a53463 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:37.747556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d3b864-ff02-4fd7-b531-c4e7d8a53463 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:37.756973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 604/1201] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:38.849987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65b8881-88b0-4e3f-9257-a1185ad142d3 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:38.850549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65b8881-88b0-4e3f-9257-a1185ad142d3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:38.850549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65b8881-88b0-4e3f-9257-a1185ad142d3 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:38.851892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 598/1202] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 20:03:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:38.856843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd43d199-fc8e-4b35-ba80-b0fa6ad4f8d8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:38.857268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd43d199-fc8e-4b35-ba80-b0fa6ad4f8d8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:38.857401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd43d199-fc8e-4b35-ba80-b0fa6ad4f8d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:38.876261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bd43d199-fc8e-4b35-ba80-b0fa6ad4f8d8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:38.877473 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bd43d199-fc8e-4b35-ba80-b0fa6ad4f8d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:38.885060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 605/1203] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 20:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:03:39.038205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65b8881-88b0-4e3f-9257-a1185ad142d3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:39.052977 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65b8881-88b0-4e3f-9257-a1185ad142d3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:39.059126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a004c5a7-2f10-4a23-bbe4-9f293a64c04c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:39.059865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a004c5a7-2f10-4a23-bbe4-9f293a64c04c service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:39.060152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a004c5a7-2f10-4a23-bbe4-9f293a64c04c service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:39.060434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a004c5a7-2f10-4a23-bbe4-9f293a64c04c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:39.068427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a004c5a7-2f10-4a23-bbe4-9f293a64c04c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:39.070054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a004c5a7-2f10-4a23-bbe4-9f293a64c04c service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:39.080645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 599/1204] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:43.290086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afc7295-e7ec-4a46-acee-313a40212f99 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:43.290619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afc7295-e7ec-4a46-acee-313a40212f99 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:43.290619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afc7295-e7ec-4a46-acee-313a40212f99 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:43.291950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 606/1205] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 20:03:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:43.297268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940cc8f-69cb-4378-b5d6-3652e2ea9d57 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:43.297713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940cc8f-69cb-4378-b5d6-3652e2ea9d57 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:43.297896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940cc8f-69cb-4378-b5d6-3652e2ea9d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:43.317447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5940cc8f-69cb-4378-b5d6-3652e2ea9d57 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:43.318662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5940cc8f-69cb-4378-b5d6-3652e2ea9d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:43.327679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 600/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 20:03:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:43.632182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afc7295-e7ec-4a46-acee-313a40212f99 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:43.633924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5afc7295-e7ec-4a46-acee-313a40212f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:43.713716 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afc7295-e7ec-4a46-acee-313a40212f99 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:43.718387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9270cf03-757c-4daa-a843-60c25b5c7b17 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:43.719001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9270cf03-757c-4daa-a843-60c25b5c7b17 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:43.719287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9270cf03-757c-4daa-a843-60c25b5c7b17 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:43.719567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9270cf03-757c-4daa-a843-60c25b5c7b17 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:43.728395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9270cf03-757c-4daa-a843-60c25b5c7b17 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:43.729866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9270cf03-757c-4daa-a843-60c25b5c7b17 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:43.740279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 607/1207] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:45.893238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b130d5-c33f-4c4a-8139-fa52aa94e1a2 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:45.893823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b130d5-c33f-4c4a-8139-fa52aa94e1a2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:45.893823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b130d5-c33f-4c4a-8139-fa52aa94e1a2 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:45.895143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 601/1208] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 20:03:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:45.900445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3b170d-4313-412e-be81-9fe31523924e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:45.900832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3b170d-4313-412e-be81-9fe31523924e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:45.901038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3b170d-4313-412e-be81-9fe31523924e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:45.925941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0a3b170d-4313-412e-be81-9fe31523924e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:45.927488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0a3b170d-4313-412e-be81-9fe31523924e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:45.935580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 608/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 20:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:46.223857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b130d5-c33f-4c4a-8139-fa52aa94e1a2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:46.229269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=75 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:03:46.235823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b130d5-c33f-4c4a-8139-fa52aa94e1a2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:46.241921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3402b71-9660-4784-a947-c9f0a4c07faa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:46.241921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3402b71-9660-4784-a947-c9f0a4c07faa service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:46.241921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3402b71-9660-4784-a947-c9f0a4c07faa service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:46.241921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3402b71-9660-4784-a947-c9f0a4c07faa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:46.249944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3402b71-9660-4784-a947-c9f0a4c07faa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:46.251402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3402b71-9660-4784-a947-c9f0a4c07faa service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:46.261393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 602/1210] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:50.040208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74120f75-5715-41e5-9a29-80d08cfcff53 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:50.040632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74120f75-5715-41e5-9a29-80d08cfcff53 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:50.040774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74120f75-5715-41e5-9a29-80d08cfcff53 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:50.042215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 609/1211] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 20:03:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:50.047558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdfb1ae-efe6-47fb-9bdd-90591e39304d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:50.047970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdfb1ae-efe6-47fb-9bdd-90591e39304d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:50.048141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdfb1ae-efe6-47fb-9bdd-90591e39304d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:50.067075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0cdfb1ae-efe6-47fb-9bdd-90591e39304d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:50.068262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0cdfb1ae-efe6-47fb-9bdd-90591e39304d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:50.076605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 603/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 20:03:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:03:50.382529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74120f75-5715-41e5-9a29-80d08cfcff53 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:50.384273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-74120f75-5715-41e5-9a29-80d08cfcff53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:50.450884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:03:50.464950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74120f75-5715-41e5-9a29-80d08cfcff53 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:50.468699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b5cc2a-ce28-4cf7-a56b-faf0aa147893 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:50.469313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5cc2a-ce28-4cf7-a56b-faf0aa147893 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:50.469630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5cc2a-ce28-4cf7-a56b-faf0aa147893 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:50.469913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5cc2a-ce28-4cf7-a56b-faf0aa147893 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:50.478824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b5cc2a-ce28-4cf7-a56b-faf0aa147893 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:50.480737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b5cc2a-ce28-4cf7-a56b-faf0aa147893 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:50.490255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 610/1213] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:03:52.731684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556719e9-8373-4dd3-829d-f1ca275d096c None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:52.732224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556719e9-8373-4dd3-829d-f1ca275d096c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:52.732224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556719e9-8373-4dd3-829d-f1ca275d096c None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:52.733667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 604/1214] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 20:03:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:03:52.738869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9cf1d1-0976-46dd-93db-fe3d299c8407 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:52.739270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9cf1d1-0976-46dd-93db-fe3d299c8407 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:52.739436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9cf1d1-0976-46dd-93db-fe3d299c8407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:52.759392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2e9cf1d1-0976-46dd-93db-fe3d299c8407 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:03:52.760589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2e9cf1d1-0976-46dd-93db-fe3d299c8407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:52.769164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 611/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 20:03:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:03:53.068442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556719e9-8373-4dd3-829d-f1ca275d096c None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:03:53.070145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-556719e9-8373-4dd3-829d-f1ca275d096c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:03:53.083355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556719e9-8373-4dd3-829d-f1ca275d096c None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:03:53.086832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef58589-0e7b-479d-bcfa-ce5ee8e57772 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:03:53.087419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef58589-0e7b-479d-bcfa-ce5ee8e57772 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:03:53.087701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef58589-0e7b-479d-bcfa-ce5ee8e57772 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:03:53.087975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef58589-0e7b-479d-bcfa-ce5ee8e57772 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:03:53.096921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef58589-0e7b-479d-bcfa-ce5ee8e57772 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:03:53.098364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef58589-0e7b-479d-bcfa-ce5ee8e57772 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:03:53.108237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 605/1216] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:03:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:04:02.754556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=11 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:04:03.103479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=14 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:16.707111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b74e5-f84c-4b6f-83a6-663db9ed7de4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.710094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b74e5-f84c-4b6f-83a6-663db9ed7de4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.710094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b74e5-f84c-4b6f-83a6-663db9ed7de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.741034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ca7b74e5-f84c-4b6f-83a6-663db9ed7de4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:16.742496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ca7b74e5-f84c-4b6f-83a6-663db9ed7de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:16.754706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 612/1217] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:16.765636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6308d97-9d76-437c-a69b-d3d32972dc57 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.766244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6308d97-9d76-437c-a69b-d3d32972dc57 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.766244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6308d97-9d76-437c-a69b-d3d32972dc57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.802593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b6308d97-9d76-437c-a69b-d3d32972dc57 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:16.803835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b6308d97-9d76-437c-a69b-d3d32972dc57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:16.813887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbe281-25ba-43b6-a168-e1dfb1d1b3f0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.814230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbe281-25ba-43b6-a168-e1dfb1d1b3f0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.814448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbe281-25ba-43b6-a168-e1dfb1d1b3f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.818174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 606/1218] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:16.822916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6308d97-9d76-437c-a69b-d3d32972dc57 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:16.836806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6308d97-9d76-437c-a69b-d3d32972dc57 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:16.850019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:16.850658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.850936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.851206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.851758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b9cbe281-25ba-43b6-a168-e1dfb1d1b3f0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:16.851883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:16.852156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:16.853192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:05:16.853192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 20:05:16.853192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 20:05:16.853942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b9cbe281-25ba-43b6-a168-e1dfb1d1b3f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:16.856047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da58f8f4-09fa-4092-ad9a-e784e5f6ee00 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:16.868624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 613/1219] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:16.876162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 607/1220] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:05:16.876811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ad2358-6e6b-44ca-a4c9-acc999bb7831 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.877378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ad2358-6e6b-44ca-a4c9-acc999bb7831 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.877811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ad2358-6e6b-44ca-a4c9-acc999bb7831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.885356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da8ec3-a6cc-4c59-aecb-0786a0bab3e7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.885356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da8ec3-a6cc-4c59-aecb-0786a0bab3e7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.885356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da8ec3-a6cc-4c59-aecb-0786a0bab3e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.911986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-54da8ec3-a6cc-4c59-aecb-0786a0bab3e7 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:16.913381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-54da8ec3-a6cc-4c59-aecb-0786a0bab3e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:16.916667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e5ad2358-6e6b-44ca-a4c9-acc999bb7831 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:16.918040 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e5ad2358-6e6b-44ca-a4c9-acc999bb7831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:16.925635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 608/1221] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:16.926741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54da8ec3-a6cc-4c59-aecb-0786a0bab3e7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:16.937715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54da8ec3-a6cc-4c59-aecb-0786a0bab3e7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:16.946193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3975cb8-1051-42d8-a575-e8fba638e64e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:16.946373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 614/1222] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:16.947000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3975cb8-1051-42d8-a575-e8fba638e64e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.947293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3975cb8-1051-42d8-a575-e8fba638e64e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.947568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf31a6-5c4a-4653-a959-cf23002ebaf8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:16.947686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3975cb8-1051-42d8-a575-e8fba638e64e admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.947857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf31a6-5c4a-4653-a959-cf23002ebaf8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:16.948123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf31a6-5c4a-4653-a959-cf23002ebaf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:16.948239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3975cb8-1051-42d8-a575-e8fba638e64e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:16.950690 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3975cb8-1051-42d8-a575-e8fba638e64e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:16.993406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8fcf31a6-5c4a-4653-a959-cf23002ebaf8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:16.994656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8fcf31a6-5c4a-4653-a959-cf23002ebaf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.004237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 615/1223] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.005059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf31a6-5c4a-4653-a959-cf23002ebaf8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.019490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf31a6-5c4a-4653-a959-cf23002ebaf8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.024746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7fd61f-c8c8-4cae-9263-318a86e92251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.029051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7fd61f-c8c8-4cae-9263-318a86e92251 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.029051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7fd61f-c8c8-4cae-9263-318a86e92251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.029051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7fd61f-c8c8-4cae-9263-318a86e92251 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.029051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7fd61f-c8c8-4cae-9263-318a86e92251 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:17.029051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7fd61f-c8c8-4cae-9263-318a86e92251 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.029051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7fd61f-c8c8-4cae-9263-318a86e92251 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.042099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 616/1224] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:17.045429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127e644-bb0d-457f-9d25-762899a5696a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.045429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127e644-bb0d-457f-9d25-762899a5696a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.045429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127e644-bb0d-457f-9d25-762899a5696a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.067195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6127e644-bb0d-457f-9d25-762899a5696a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.068439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6127e644-bb0d-457f-9d25-762899a5696a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.078235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 617/1225] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.080113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21276a-f984-414d-aec1-5f756d420340 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.080399 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21276a-f984-414d-aec1-5f756d420340 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.080674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21276a-f984-414d-aec1-5f756d420340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.100562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 609/1226] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:16 2022] POST /identity/v3/projects => generated 383 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:17.102931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f94b0d-1ee8-4f0a-81cb-33282ade0b8b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.103036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6d21276a-f984-414d-aec1-5f756d420340 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.103122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f94b0d-1ee8-4f0a-81cb-33282ade0b8b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.103122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f94b0d-1ee8-4f0a-81cb-33282ade0b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.103938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6d21276a-f984-414d-aec1-5f756d420340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.123037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b9f94b0d-1ee8-4f0a-81cb-33282ade0b8b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.124229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b9f94b0d-1ee8-4f0a-81cb-33282ade0b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.165736 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 618/1227] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.167041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03ee7e-9823-4afb-b398-67ce57f7ce1a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.167358 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03ee7e-9823-4afb-b398-67ce57f7ce1a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.167575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03ee7e-9823-4afb-b398-67ce57f7ce1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.184031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 610/1228] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.185575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f94b0d-1ee8-4f0a-81cb-33282ade0b8b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.188113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3e03ee7e-9823-4afb-b398-67ce57f7ce1a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.189335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3e03ee7e-9823-4afb-b398-67ce57f7ce1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.197496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 619/1229] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.198708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9069445-9a64-40ca-ae16-1abe283b74ff None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.199003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9069445-9a64-40ca-ae16-1abe283b74ff None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.199106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f94b0d-1ee8-4f0a-81cb-33282ade0b8b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.199300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9069445-9a64-40ca-ae16-1abe283b74ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.204136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.205001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.205273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.205589 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.206211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.208094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.219599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b9069445-9a64-40ca-ae16-1abe283b74ff None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.220796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b9069445-9a64-40ca-ae16-1abe283b74ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.228717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 620/1230] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.232865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b35ed5-bd44-4067-8f2c-53d1cf4bb500 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.232865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b35ed5-bd44-4067-8f2c-53d1cf4bb500 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.232865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b35ed5-bd44-4067-8f2c-53d1cf4bb500 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.250322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-88b35ed5-bd44-4067-8f2c-53d1cf4bb500 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.251443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-88b35ed5-bd44-4067-8f2c-53d1cf4bb500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.259255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 621/1231] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.260412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae183f46-53f1-46d1-bf54-7d93c5b94093 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.261629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae183f46-53f1-46d1-bf54-7d93c5b94093 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.261629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae183f46-53f1-46d1-bf54-7d93c5b94093 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.280675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ae183f46-53f1-46d1-bf54-7d93c5b94093 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.281881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ae183f46-53f1-46d1-bf54-7d93c5b94093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.289645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 622/1232] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.292299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae183f46-53f1-46d1-bf54-7d93c5b94093 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.300216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae183f46-53f1-46d1-bf54-7d93c5b94093 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.304705 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.305279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.305591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.305868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.306477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.308366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.317635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 611/1233] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/projects => generated 393 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:17.328598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.380103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04799a6-396c-490c-8afb-dcbaee74b7be admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.384806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02f4914-5221-4d46-8ff2-a85f964b3a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.385441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02f4914-5221-4d46-8ff2-a85f964b3a42 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.385731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02f4914-5221-4d46-8ff2-a85f964b3a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.386009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02f4914-5221-4d46-8ff2-a85f964b3a42 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.386652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02f4914-5221-4d46-8ff2-a85f964b3a42 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:17.386956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02f4914-5221-4d46-8ff2-a85f964b3a42 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.388664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02f4914-5221-4d46-8ff2-a85f964b3a42 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.399961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 612/1234] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:17.404153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 623/1235] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:17.404299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab824c1-d2cc-4012-9be6-f13301e4b054 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.404299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab824c1-d2cc-4012-9be6-f13301e4b054 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.404485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.404584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab824c1-d2cc-4012-9be6-f13301e4b054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.417408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758e6ff-8877-4266-9e84-894bfa03c951 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.423148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1810f25b-49dd-4efa-baac-b7e2daf8a5df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.423761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810f25b-49dd-4efa-baac-b7e2daf8a5df admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.424044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810f25b-49dd-4efa-baac-b7e2daf8a5df admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.424327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1810f25b-49dd-4efa-baac-b7e2daf8a5df admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.424990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1810f25b-49dd-4efa-baac-b7e2daf8a5df admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:17.425094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-cab824c1-d2cc-4012-9be6-f13301e4b054 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.425278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1810f25b-49dd-4efa-baac-b7e2daf8a5df admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.426323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-cab824c1-d2cc-4012-9be6-f13301e4b054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.426912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1810f25b-49dd-4efa-baac-b7e2daf8a5df admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.434308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 613/1236] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.436072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0143c-50c1-4886-b13d-07f099b6b4e8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.436353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0143c-50c1-4886-b13d-07f099b6b4e8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.436634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da0143c-50c1-4886-b13d-07f099b6b4e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.440243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 624/1237] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:17.441668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bb8be7-bc3d-4c99-8ff7-0629a81e1216 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.441910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bb8be7-bc3d-4c99-8ff7-0629a81e1216 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.442187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bb8be7-bc3d-4c99-8ff7-0629a81e1216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.458048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3da0143c-50c1-4886-b13d-07f099b6b4e8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.459259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3da0143c-50c1-4886-b13d-07f099b6b4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.465108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-23bb8be7-bc3d-4c99-8ff7-0629a81e1216 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:17.466836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-23bb8be7-bc3d-4c99-8ff7-0629a81e1216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:17.467338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 614/1238] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.470342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da0143c-50c1-4886-b13d-07f099b6b4e8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.474837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 625/1239] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:17.475413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:17.475825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bb8be7-bc3d-4c99-8ff7-0629a81e1216 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.481127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da0143c-50c1-4886-b13d-07f099b6b4e8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.488525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.488738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bb8be7-bc3d-4c99-8ff7-0629a81e1216 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.488845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.488845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.488845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.488845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:17.488845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.490152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.491910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.492525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.492812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.493095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.493765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.495675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.505700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 615/1240] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:17.506645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.516869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1d77ee-2774-40a0-bd93-0892d45515f5 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.521563 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.522147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.522427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.522697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.523324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.524929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.535698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 616/1241] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:17.537269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.545740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdea1e53-406d-4b1c-a6e0-289b532d7887 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.550336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.550898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.551177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.551448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.552050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.553937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.578467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 626/1242] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/users => generated 297 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:17.583404 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.588878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de38383-4572-40d5-9fb0-1d73229a87fb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.593658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.594240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.594520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.594795 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.595425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:17.595731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.597309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.609834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 627/1243] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:17.610617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.622853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b50248-56ea-4556-9a99-23103427712a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.628426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.628934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.629221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.629599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.630201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.631240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:05:17.631240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 20:05:17.631240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 20:05:17.634794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 617/1244] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/projects => generated 405 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:17.634794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.634985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.695357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a75a8-52ec-4ff2-b6fb-9a868443402c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.700061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.700687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.700970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.701249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.701925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:17.702235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.703885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.704360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:17.715223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 628/1245] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/projects => generated 377 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:17.715223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.716996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 618/1246] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:17.717858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.727565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be01af11-16ce-4a3f-89d7-3f89019bd90f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.786024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44fc23e-9126-4320-81c0-a37cbd40856a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.786234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.786681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.786878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.787149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.787777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.789806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.791134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.791749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.792036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.792343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.792993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:17.793297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.795475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.807430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 629/1247] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:17.808242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.820246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e65866a-c093-4a2c-a2e0-ec701a4ed0bb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.824896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.825701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.825980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.826268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.826903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.828487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.839147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 630/1248] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:17.840816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.849373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda8c7d-c3ba-41d3-81ee-d15ab6afa503 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.854235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.854818 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.855097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.855372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.855994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.858372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.862113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 619/1249] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/projects => generated 377 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:17.862113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.871889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 631/1250] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:17.873306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:17.883150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db275c-911b-4d24-a1a9-1c816837c119 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.905207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:17.928978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b9bdd2-4028-4ce1-957f-32b786f9cff4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:17.933746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.935013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.935535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.935801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.936422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.938357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:17.941602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:17.941853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:17.942238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:17.942620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:17.943318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:17.945255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.054087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 620/1251] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/projects => generated 389 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.054900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.082638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 632/1252] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:17 2022] POST /identity/v3/users => generated 303 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.082638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.094713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b108e52b-c0d8-4846-98b0-81855dfc35b3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.118877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb12e0d-d7db-4fe8-9aa1-e2522530f620 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.124234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.124844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.125142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.125444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.126074 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.127099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:05:18.127099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 20:05:18.127099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 20:05:18.128343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.153724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.154401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.154678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.154955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.155588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.157642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.162126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:18.207562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 621/1253] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/users => generated 289 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.208373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.217900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b437137-0cbb-453a-a82f-8c39ff857960 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.223021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.223645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.223932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.224208 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.224830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.226859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.237842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 633/1254] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/projects => generated 383 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.237842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.297071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c764b6b-b4b4-4e81-910f-0c2cb2ffad26 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.301725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.302389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.306051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.306051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] PATH_INFO: `/v3/projects/940ce72c42534c5196dfb909cc2a8d54/users/724baaa4aa834e609fe4f513f9fc0b4b/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.318204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] RBAC: Authorizing `identity:create_grant(project_id=940ce72c42534c5196dfb909cc2a8d54, user_id=724baaa4aa834e609fe4f513f9fc0b4b, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.320867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.322649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 622/1255] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/projects => generated 393 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.327183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.383893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 634/1256] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:18 2022] PUT /identity/v3/projects/940ce72c42534c5196dfb909cc2a8d54/users/724baaa4aa834e609fe4f513f9fc0b4b/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:18.384765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.393010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01822f6f-a6be-4880-97b3-dd25aa043560 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.426938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:18.448197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005b7342-c445-41d7-8fb0-cf53fb53c384 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.453544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.454131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.454412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.454784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.454956 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.455371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.455580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.455689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.455937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.456586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.457179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.458553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.468104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 635/1257] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.468914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.479016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ef3c3-ea66-4ca3-aa8f-1bd06833d95b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.483726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.484318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.484601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.493696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.493696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.493696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.572918 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 623/1258] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/users => generated 289 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.573849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.585153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8858c8-c03f-4699-ae2c-6cae16e9fa0d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.589958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.590542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.591984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.592265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.592899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.594573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.601448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 636/1259] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/users => generated 295 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.605258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.606949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 624/1260] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.608232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.615600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e4bf13-8a52-4608-bd33-2fb94ca68e28 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.618302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3678e80-4558-439f-869f-054c4e3d4a7f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.620193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.620800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.621086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.621420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.622065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.623655 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2d4408-2c84-40dd-a0d6-a451bcfb313e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.623843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.624259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d4408-2c84-40dd-a0d6-a451bcfb313e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.624548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d4408-2c84-40dd-a0d6-a451bcfb313e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.624825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d4408-2c84-40dd-a0d6-a451bcfb313e admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.625514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2d4408-2c84-40dd-a0d6-a451bcfb313e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.628580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2d4408-2c84-40dd-a0d6-a451bcfb313e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.635519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 637/1261] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.636320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.646871 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe818ce-39a7-498d-925b-4282cd004f09 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.651743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.654913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.654913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.654913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.654913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.656518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.710985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 625/1262] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.712360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a98d9b-fa61-4b06-aa18-84e016872611 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.712719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a98d9b-fa61-4b06-aa18-84e016872611 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.713002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a98d9b-fa61-4b06-aa18-84e016872611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.734442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c2a98d9b-fa61-4b06-aa18-84e016872611 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:18.735652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c2a98d9b-fa61-4b06-aa18-84e016872611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:18.743736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 626/1263] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:18.746626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a98d9b-fa61-4b06-aa18-84e016872611 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.754698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a98d9b-fa61-4b06-aa18-84e016872611 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.759386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.760027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.760310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.760584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] PATH_INFO: `/v3/projects/bb6cdda7411a4d16880f5f52725afdec/users/ad7ecfe96cd944a29861ce8f66fc6eb0/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.771132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 638/1264] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:18 2022] POST /identity/v3/users => generated 297 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:18.771132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.777219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb6cdda7411a4d16880f5f52725afdec, user_id=ad7ecfe96cd944a29861ce8f66fc6eb0, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.779003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd228ea-a6fa-43b8-ae04-620b3e301d5b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.780373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.784197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.784802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.785087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.785401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.786045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.787723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.798762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 639/1265] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.799548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.809636 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50afd3a1-f421-4621-ae43-e37250939631 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.814793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.815381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.815667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.815943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.816568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.818176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.819426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:18.829863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 640/1266] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.830452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.839595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f54980-f781-42a2-96e7-9c6a54ba83e8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.844402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.844981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.845267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.845574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.846196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.847730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 627/1267] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:18 2022] PUT /identity/v3/projects/bb6cdda7411a4d16880f5f52725afdec/users/ad7ecfe96cd944a29861ce8f66fc6eb0/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:18.848183 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.848421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.861437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 641/1268] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.861437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.868145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292833ca-a1bc-4c50-a189-c166f667d993 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.921805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b1bf9b-f819-4702-b18f-499857d8a5a3 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.926364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.926515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.926975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.927078 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.927259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.927368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.927534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.927652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.928174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.928291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.929829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.929955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.940543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 642/1269] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.944218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 628/1270] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.944379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.944493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.953870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7c92da-b1dd-4889-85ec-91b39f7309ab admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.958778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.959376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.959661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.959940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.960691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.962323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:18.972583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 629/1271] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:18.973399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:18.983234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a837f67-a0bc-4cda-a3a4-fda07785707c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:18.987892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:18.988477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:18.988762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:18.989037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:18.989702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:18.991278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.000415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3b3994-db8c-43b6-9559-f70df48d755e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.002577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 630/1272] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:19.003460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.004535 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8529723e-85ac-4408-b0e8-4d27bb189916 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.005149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8529723e-85ac-4408-b0e8-4d27bb189916 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.005533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8529723e-85ac-4408-b0e8-4d27bb189916 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.006101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8529723e-85ac-4408-b0e8-4d27bb189916 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.014621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c051a-faeb-4691-9cc3-f62d57e6e560 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.015574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8529723e-85ac-4408-b0e8-4d27bb189916 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.017176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8529723e-85ac-4408-b0e8-4d27bb189916 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.019244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.019842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.020126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.020399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.021030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.022654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.029669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 643/1273] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:19.029669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41c4f52-d0d6-46fb-bd84-a7c96b7cdc24 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.029669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41c4f52-d0d6-46fb-bd84-a7c96b7cdc24 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.030171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41c4f52-d0d6-46fb-bd84-a7c96b7cdc24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.035984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 631/1274] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:19.036819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.048883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a798f12-fbf0-4c55-935f-6e6dba7574c2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.054576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.055374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.055663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.055940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] PATH_INFO: `/v3/projects/51a485932b6b4e0086066885364eb974/users/c7931041aa5a4546bfeaf5e8cfe50f41/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.061579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a41c4f52-d0d6-46fb-bd84-a7c96b7cdc24 None None] MFA Rules not processed for user `ad7ecfe96cd944a29861ce8f66fc6eb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:19.062804 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a41c4f52-d0d6-46fb-bd84-a7c96b7cdc24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:19.072305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=51a485932b6b4e0086066885364eb974, user_id=c7931041aa5a4546bfeaf5e8cfe50f41, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.075083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.127211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 644/1275] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:19 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:19.128119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41c4f52-d0d6-46fb-bd84-a7c96b7cdc24 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.138144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41c4f52-d0d6-46fb-bd84-a7c96b7cdc24 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.143629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.144237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.144543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.144821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.145538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.148312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.159209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 645/1276] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:19.163110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.168234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 632/1277] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:19 2022] PUT /identity/v3/projects/51a485932b6b4e0086066885364eb974/users/c7931041aa5a4546bfeaf5e8cfe50f41/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:19.169095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.171389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:19.176859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb94951-36a6-4ff5-b71c-79b584a8f27e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.177187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:19.238187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.239074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.239359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.239640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] PATH_INFO: `/v3/projects/40fa4f575da04bc39b8df1e74fb77035/users/1b3396bc67ae4f6c9c8808653fef4676/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.241861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b683c84c-308d-48a5-b918-214f1a3a5d21 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.246542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.247185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.247472 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.247802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.248809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.250544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.254844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] RBAC: Authorizing `identity:create_grant(project_id=40fa4f575da04bc39b8df1e74fb77035, user_id=1b3396bc67ae4f6c9c8808653fef4676, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.257598 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.260724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 633/1278] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:19 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:19.261611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.272688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea258d90-d32c-4c4d-b855-a48c5370851f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.277567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.278214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.278496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.278773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] PATH_INFO: `/v3/projects/8b163bdfc7544a3d833bd69dc027c7d6/users/15db5106efbd434f99a4d102810cc7a6/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.294416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b163bdfc7544a3d833bd69dc027c7d6, user_id=15db5106efbd434f99a4d102810cc7a6, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.297237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.361334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 646/1279] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:19 2022] PUT /identity/v3/projects/40fa4f575da04bc39b8df1e74fb77035/users/1b3396bc67ae4f6c9c8808653fef4676/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:19.362392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.380764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 634/1280] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:19 2022] PUT /identity/v3/projects/8b163bdfc7544a3d833bd69dc027c7d6/users/15db5106efbd434f99a4d102810cc7a6/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:19.380764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.432017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153d9670-63d9-46fd-8bc2-c4dc089b193e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.465749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c9f9c4-5e29-4e45-8cfa-e5027ea38564 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.471826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.472967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.472967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.473211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.473875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.475881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.496524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf4c27b-43b4-4db3-b88d-a4127a931075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.497200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf4c27b-43b4-4db3-b88d-a4127a931075 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.498107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf4c27b-43b4-4db3-b88d-a4127a931075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.509225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf4c27b-43b4-4db3-b88d-a4127a931075 admin admin] PATH_INFO: `/v3/projects/34d30fbe6fa645bd9ea3d88ea5f68449/users/e0ba6bf2459d4a98a3a1f5684bdad2b2/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.512510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf4c27b-43b4-4db3-b88d-a4127a931075 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34d30fbe6fa645bd9ea3d88ea5f68449, user_id=e0ba6bf2459d4a98a3a1f5684bdad2b2, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.515899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf4c27b-43b4-4db3-b88d-a4127a931075 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.574212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 635/1281] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:19 2022] POST /identity/v3/projects => generated 393 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:19.575210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.602989 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 647/1282] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:19 2022] PUT /identity/v3/projects/34d30fbe6fa645bd9ea3d88ea5f68449/users/e0ba6bf2459d4a98a3a1f5684bdad2b2/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:19.604172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4934f7f-6b0b-40ae-b492-9bc059c012f0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.607709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4934f7f-6b0b-40ae-b492-9bc059c012f0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.607709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4934f7f-6b0b-40ae-b492-9bc059c012f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.626206 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b4934f7f-6b0b-40ae-b492-9bc059c012f0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:19.627406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b4934f7f-6b0b-40ae-b492-9bc059c012f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:19.637951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:19.653871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034aff9b-1304-4b71-ad00-7a4583e5934b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.665296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:19.704402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 648/1283] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:19.705386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4934f7f-6b0b-40ae-b492-9bc059c012f0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.715505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4934f7f-6b0b-40ae-b492-9bc059c012f0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.720363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de62dfc0-3b73-47d7-a08f-cb5e7eda715f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.721066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62dfc0-3b73-47d7-a08f-cb5e7eda715f admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.721359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62dfc0-3b73-47d7-a08f-cb5e7eda715f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.721643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62dfc0-3b73-47d7-a08f-cb5e7eda715f admin admin] PATH_INFO: `/v3/projects/ab907bf956554681beae45b8973db6e5/users/ce7f52b7f96c400fb7d5fb0453c94362/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.728460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fb3441-6b76-45d2-bfdf-ff734a679d2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.733906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fb3441-6b76-45d2-bfdf-ff734a679d2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.733906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fb3441-6b76-45d2-bfdf-ff734a679d2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.733906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fb3441-6b76-45d2-bfdf-ff734a679d2c admin admin] PATH_INFO: `/v3/projects/f6c2f7094bcf43c49ea03ea775595f51/users/f9eaff8ed7204e6e8fddbd983f1d20ee/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.736302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de62dfc0-3b73-47d7-a08f-cb5e7eda715f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab907bf956554681beae45b8973db6e5, user_id=ce7f52b7f96c400fb7d5fb0453c94362, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.738994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de62dfc0-3b73-47d7-a08f-cb5e7eda715f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.744730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fb3441-6b76-45d2-bfdf-ff734a679d2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6c2f7094bcf43c49ea03ea775595f51, user_id=f9eaff8ed7204e6e8fddbd983f1d20ee, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.747470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fb3441-6b76-45d2-bfdf-ff734a679d2c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.813382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 649/1284] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:19 2022] PUT /identity/v3/projects/ab907bf956554681beae45b8973db6e5/users/ce7f52b7f96c400fb7d5fb0453c94362/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:19.814856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c64bfed-7994-4b6c-bf0e-aa0b389907be None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.815148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c64bfed-7994-4b6c-bf0e-aa0b389907be None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.815428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c64bfed-7994-4b6c-bf0e-aa0b389907be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.834703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9c64bfed-7994-4b6c-bf0e-aa0b389907be None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:19.835908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9c64bfed-7994-4b6c-bf0e-aa0b389907be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:19.856096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 636/1285] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:19 2022] PUT /identity/v3/projects/f6c2f7094bcf43c49ea03ea775595f51/users/f9eaff8ed7204e6e8fddbd983f1d20ee/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:19.856096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5256fa-5b6d-4b58-b396-53033eb14097 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.856096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5256fa-5b6d-4b58-b396-53033eb14097 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.856841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5256fa-5b6d-4b58-b396-53033eb14097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.886067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-7d5256fa-5b6d-4b58-b396-53033eb14097 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:19.890099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7d5256fa-5b6d-4b58-b396-53033eb14097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:19.903423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 650/1286] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:19.907747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c64bfed-7994-4b6c-bf0e-aa0b389907be None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.917158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c64bfed-7994-4b6c-bf0e-aa0b389907be None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:19.957447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 637/1287] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:19.958982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.959317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.959597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.979780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:19.980967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:19.985073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f53258b-537a-42fb-a054-e11a381378fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:19.987446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53258b-537a-42fb-a054-e11a381378fa admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:19.987446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53258b-537a-42fb-a054-e11a381378fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:19.987446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53258b-537a-42fb-a054-e11a381378fa admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:19.987446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f53258b-537a-42fb-a054-e11a381378fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:19.988773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f53258b-537a-42fb-a054-e11a381378fa admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:19.989838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 638/1288] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:19.990624 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:19.991843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:20.050584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddbaaa6-dd9e-41b8-992e-7c23f7260b6a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.054256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d7fdc4-0afa-4228-b7f9-476a7077c043 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.054872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d7fdc4-0afa-4228-b7f9-476a7077c043 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.055155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d7fdc4-0afa-4228-b7f9-476a7077c043 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.055430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d7fdc4-0afa-4228-b7f9-476a7077c043 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.065161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d7fdc4-0afa-4228-b7f9-476a7077c043 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.066633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d7fdc4-0afa-4228-b7f9-476a7077c043 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.073706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 651/1289] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:19 2022] POST /identity/v3/users => generated 303 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:20.073706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9662f34b-1558-48a5-bb27-80fe87c568bb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.073706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9662f34b-1558-48a5-bb27-80fe87c568bb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.073706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9662f34b-1558-48a5-bb27-80fe87c568bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.078651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 639/1290] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:20.080238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad679c3-a6eb-466f-8ba8-953719feff45 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.080601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad679c3-a6eb-466f-8ba8-953719feff45 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.080957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad679c3-a6eb-466f-8ba8-953719feff45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.095243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9662f34b-1558-48a5-bb27-80fe87c568bb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:20.096658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9662f34b-1558-48a5-bb27-80fe87c568bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:20.106322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 652/1291] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:20.111023 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9662f34b-1558-48a5-bb27-80fe87c568bb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.113203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3ad679c3-a6eb-466f-8ba8-953719feff45 None None] MFA Rules not processed for user `f9eaff8ed7204e6e8fddbd983f1d20ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:20.114555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3ad679c3-a6eb-466f-8ba8-953719feff45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:20.121944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9662f34b-1558-48a5-bb27-80fe87c568bb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.127499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.128312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.128927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.129065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.134318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:20.142910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.144613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.160489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 653/1292] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:20.160489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.164283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:20.168982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e447a3-67c5-4d5b-904a-19d4960c068d service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.173153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.173853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.174217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.174421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.184698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.189164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.191058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 640/1293] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:20.191910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad679c3-a6eb-466f-8ba8-953719feff45 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.201940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad679c3-a6eb-466f-8ba8-953719feff45 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.202494 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 654/1294] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:20.203402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.218201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83161d-4c1f-47e0-88b2-09f73341f343 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.218313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816aa31b-3c49-49e9-8719-71ff9ff66111 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.218313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816aa31b-3c49-49e9-8719-71ff9ff66111 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.218678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816aa31b-3c49-49e9-8719-71ff9ff66111 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.218991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816aa31b-3c49-49e9-8719-71ff9ff66111 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.227872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.228629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.228773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.228985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.229690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.231311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.236011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816aa31b-3c49-49e9-8719-71ff9ff66111 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.237807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816aa31b-3c49-49e9-8719-71ff9ff66111 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.243791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 655/1295] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:20.244655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.251920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 641/1296] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:20.253578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5323f926-4d61-4368-836c-3728cb3e4fc7 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.253900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5323f926-4d61-4368-836c-3728cb3e4fc7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.254158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5323f926-4d61-4368-836c-3728cb3e4fc7 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.255447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 642/1297] 10.222.0.66 () {54 vars in 961 bytes} [Fri Oct 28 20:05:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:05:20.256130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5323f926-4d61-4368-836c-3728cb3e4fc7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.258960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a3fa5b-ecab-4fed-a3d7-4a0a6a5d9614 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.263344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659604e4-7bc4-439b-9a0b-625428368340 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.264286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659604e4-7bc4-439b-9a0b-625428368340 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.265120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659604e4-7bc4-439b-9a0b-625428368340 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.265120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659604e4-7bc4-439b-9a0b-625428368340 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.269988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5323f926-4d61-4368-836c-3728cb3e4fc7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.275640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659604e4-7bc4-439b-9a0b-625428368340 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.276259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.277411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.277411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.277627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659604e4-7bc4-439b-9a0b-625428368340 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.277717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] PATH_INFO: `/v3/projects/c85093aea4804fb6bf7e9de10f17e434/users/2e482f31f548492ead2875091dbc1e70/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.289559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 656/1298] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:20.290892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04da4260-63b7-4d38-ad57-867c64f99828 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.291322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04da4260-63b7-4d38-ad57-867c64f99828 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.291602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04da4260-63b7-4d38-ad57-867c64f99828 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.309917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=c85093aea4804fb6bf7e9de10f17e434, user_id=2e482f31f548492ead2875091dbc1e70, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.316620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.337993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-04da4260-63b7-4d38-ad57-867c64f99828 None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:20.339327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-04da4260-63b7-4d38-ad57-867c64f99828 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:20.414878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 657/1299] 10.222.0.66 () {58 vars in 1058 bytes} [Fri Oct 28 20:05:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:20.427126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04da4260-63b7-4d38-ad57-867c64f99828 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.429825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 643/1300] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:20 2022] PUT /identity/v3/projects/c85093aea4804fb6bf7e9de10f17e434/users/2e482f31f548492ead2875091dbc1e70/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:20.436488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.440446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04da4260-63b7-4d38-ad57-867c64f99828 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.577977 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33bb53-6767-4871-a2b5-eb50c6baf9ca admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.578220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206fc053-65bb-4e6c-9aa7-b76189fb9113 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.579014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206fc053-65bb-4e6c-9aa7-b76189fb9113 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.579304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206fc053-65bb-4e6c-9aa7-b76189fb9113 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.579836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206fc053-65bb-4e6c-9aa7-b76189fb9113 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.584417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.585246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.585628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.590295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.590295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.590295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.608185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 644/1301] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:20.612051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.637998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d721c3be-1d4d-4dcb-99c9-a622411743b8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:20.658203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:20.659389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.660189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.661445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] PATH_INFO: `/v3/projects/c85093aea4804fb6bf7e9de10f17e434/users/2e482f31f548492ead2875091dbc1e70/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.667950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c85093aea4804fb6bf7e9de10f17e434, user_id=2e482f31f548492ead2875091dbc1e70, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.672338 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206fc053-65bb-4e6c-9aa7-b76189fb9113 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:20.674702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206fc053-65bb-4e6c-9aa7-b76189fb9113 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.677218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:20.689743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 658/1302] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:05:20.716798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 645/1303] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:20 2022] PUT /identity/v3/projects/c85093aea4804fb6bf7e9de10f17e434/users/2e482f31f548492ead2875091dbc1e70/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:20.721153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88ce33-e9c9-48cd-aa21-3ee942ed7afa None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:20.721663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88ce33-e9c9-48cd-aa21-3ee942ed7afa None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:20.722165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88ce33-e9c9-48cd-aa21-3ee942ed7afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:20.774259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6f88ce33-e9c9-48cd-aa21-3ee942ed7afa None None] MFA Rules not processed for user `2e482f31f548492ead2875091dbc1e70`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:20.777274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6f88ce33-e9c9-48cd-aa21-3ee942ed7afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:20.889505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 659/1304] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:20 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:20.903161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:20.922070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:20.997168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3afde53-ca64-42c6-9bff-376686f57013 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:21.007571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5566941c-5f33-4175-837f-3d19b11b5552 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:21.008338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5566941c-5f33-4175-837f-3d19b11b5552 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:21.018737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5566941c-5f33-4175-837f-3d19b11b5552 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:21.019074 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5566941c-5f33-4175-837f-3d19b11b5552 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:21.039000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5566941c-5f33-4175-837f-3d19b11b5552 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:21.040523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5566941c-5f33-4175-837f-3d19b11b5552 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:21.054832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802a8e81-d039-4969-8ac9-c66f6df2055d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:21.055214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802a8e81-d039-4969-8ac9-c66f6df2055d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:21.055569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802a8e81-d039-4969-8ac9-c66f6df2055d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:21.056226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 646/1305] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:20 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:05:21.058849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 660/1306] 10.222.0.66 () {54 vars in 912 bytes} [Fri Oct 28 20:05:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 20:05:21.066937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a246b30-da95-4297-8999-3aa5424df04f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:21.067397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a246b30-da95-4297-8999-3aa5424df04f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:21.068250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a246b30-da95-4297-8999-3aa5424df04f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:21.083921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:21.092029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3a246b30-da95-4297-8999-3aa5424df04f None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:21.093531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3a246b30-da95-4297-8999-3aa5424df04f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:21.180966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802a8e81-d039-4969-8ac9-c66f6df2055d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:21.202706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802a8e81-d039-4969-8ac9-c66f6df2055d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:21.208846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8764974d-7c62-4785-a24e-c5d8c5661db3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:21.209767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764974d-7c62-4785-a24e-c5d8c5661db3 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:21.210230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764974d-7c62-4785-a24e-c5d8c5661db3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:21.210709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764974d-7c62-4785-a24e-c5d8c5661db3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:21.241014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 647/1307] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:05:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:21.265794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a246b30-da95-4297-8999-3aa5424df04f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:21.280810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a246b30-da95-4297-8999-3aa5424df04f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:21.283799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764974d-7c62-4785-a24e-c5d8c5661db3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:21.286874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764974d-7c62-4785-a24e-c5d8c5661db3 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:21.286958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:21.286958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:21.286958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:21.286958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:21.297153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 661/1308] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:21 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:21.330750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:21.332283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:21.346213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 648/1309] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:22.765198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619244bd-9c37-4148-b38d-eb7c1fbb4f7c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:22.765843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619244bd-9c37-4148-b38d-eb7c1fbb4f7c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:22.765843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619244bd-9c37-4148-b38d-eb7c1fbb4f7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:22.840769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-619244bd-9c37-4148-b38d-eb7c1fbb4f7c None None] MFA Rules not processed for user `c7931041aa5a4546bfeaf5e8cfe50f41`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:22.842286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-619244bd-9c37-4148-b38d-eb7c1fbb4f7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:23.014634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:23.035114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 662/1310] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:22 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:23.039718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:23.133979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85133641-865c-41ff-9d02-2f268bfbb699 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:23.150800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:23.153598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:23.153598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:23.153598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:23.153598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:23.154862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:23.316776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 649/1311] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:23 2022] POST /identity/v3/projects => generated 403 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:05:23.318596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619244bd-9c37-4148-b38d-eb7c1fbb4f7c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:23.336789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619244bd-9c37-4148-b38d-eb7c1fbb4f7c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:23.434558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:23.435952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:23.436252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:23.436660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:23.437363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:23.439977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:23.519164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 663/1312] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:23 2022] POST /identity/v3/users => generated 308 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:23.520420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:23.542312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948d5062-19cf-4c8c-bd36-ab210287957d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:23.552072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:23.552712 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:23.557014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:23.557014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:23.557014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:23.557420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:23.587185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 650/1313] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:23 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:23.588728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:23.608793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08708156-4551-4b0b-bcee-7ec15ed028b7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:23.643639 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:23.645947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:23.646895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:23.646895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] PATH_INFO: `/v3/projects/79ab60d1d83e4e59bc92904965ccbf37/users/ca20502cea544288803af97a4da45d8c/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:23.692345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79ab60d1d83e4e59bc92904965ccbf37, user_id=ca20502cea544288803af97a4da45d8c, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:23.700225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:23.813112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 664/1314] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:23 2022] PUT /identity/v3/projects/79ab60d1d83e4e59bc92904965ccbf37/users/ca20502cea544288803af97a4da45d8c/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:23.814197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:23.833371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac422f7a-6757-4e94-a329-9378afa5a998 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:23.960184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba73a47-87e6-4daf-a528-02fa5f44e26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:23.961023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba73a47-87e6-4daf-a528-02fa5f44e26f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:23.963876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba73a47-87e6-4daf-a528-02fa5f44e26f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:23.964438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba73a47-87e6-4daf-a528-02fa5f44e26f admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:23.965634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba73a47-87e6-4daf-a528-02fa5f44e26f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:23.968290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba73a47-87e6-4daf-a528-02fa5f44e26f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:23.970130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:23.988431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:23.995905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 651/1315] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:23 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:24.004024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c15e804-c6c1-4161-8eda-c7f9bb264445 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:24.010355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01411e2-701d-4906-8d87-e1198b66cd5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:24.011844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01411e2-701d-4906-8d87-e1198b66cd5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:24.012367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01411e2-701d-4906-8d87-e1198b66cd5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:24.012847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01411e2-701d-4906-8d87-e1198b66cd5d admin admin] PATH_INFO: `/v3/projects/79ab60d1d83e4e59bc92904965ccbf37/users/ca20502cea544288803af97a4da45d8c/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:24.015611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01411e2-701d-4906-8d87-e1198b66cd5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=79ab60d1d83e4e59bc92904965ccbf37, user_id=ca20502cea544288803af97a4da45d8c, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:24.019657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01411e2-701d-4906-8d87-e1198b66cd5d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:24.045080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 665/1316] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:23 2022] PUT /identity/v3/projects/79ab60d1d83e4e59bc92904965ccbf37/users/ca20502cea544288803af97a4da45d8c/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:24.196359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be4a88-6536-4dc5-bffe-24a46fc7ec63 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:24.203360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be4a88-6536-4dc5-bffe-24a46fc7ec63 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:24.203360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be4a88-6536-4dc5-bffe-24a46fc7ec63 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:24.203360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 652/1317] 10.222.0.66 () {54 vars in 961 bytes} [Fri Oct 28 20:05:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:05:24.207285 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da5132-71f9-4163-9ddb-ea99afc5673e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:24.207593 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da5132-71f9-4163-9ddb-ea99afc5673e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:24.207858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da5132-71f9-4163-9ddb-ea99afc5673e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:24.228544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-40da5132-71f9-4163-9ddb-ea99afc5673e None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:24.229812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-40da5132-71f9-4163-9ddb-ea99afc5673e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:24.328649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 666/1318] 10.222.0.66 () {58 vars in 1058 bytes} [Fri Oct 28 20:05:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:24.335586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a40df4-a7a4-4f97-9433-3c55a86e92f8 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:24.336032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a40df4-a7a4-4f97-9433-3c55a86e92f8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:24.336425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a40df4-a7a4-4f97-9433-3c55a86e92f8 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:24.337897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 653/1319] 10.222.0.66 () {54 vars in 912 bytes} [Fri Oct 28 20:05:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:05:24.345394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaf08d8-6ce6-438c-830c-50f13322b934 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:24.345746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaf08d8-6ce6-438c-830c-50f13322b934 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:24.346527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaf08d8-6ce6-438c-830c-50f13322b934 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:24.380514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9aaf08d8-6ce6-438c-830c-50f13322b934 None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:24.384290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9aaf08d8-6ce6-438c-830c-50f13322b934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:24.415647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 667/1320] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:05:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:24.418661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a40df4-a7a4-4f97-9433-3c55a86e92f8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:24.519648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a40df4-a7a4-4f97-9433-3c55a86e92f8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:24.523731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:24.524622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:24.524915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:24.525195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:24.546451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:24.549292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:24.567750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 654/1321] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:28.181061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7876b880-c9af-4199-98cf-24c18eed1de4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:28.181061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7876b880-c9af-4199-98cf-24c18eed1de4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:28.181061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7876b880-c9af-4199-98cf-24c18eed1de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:28.267336 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7876b880-c9af-4199-98cf-24c18eed1de4 None None] MFA Rules not processed for user `ca20502cea544288803af97a4da45d8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:28.268809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7876b880-c9af-4199-98cf-24c18eed1de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:28.311111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:28.391634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 668/1322] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:28 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:30.316656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:30.474720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4de3d2-1184-4469-bf93-e6d27d281cc7 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:30.485053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:30.485757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:30.486302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:30.486771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] PATH_INFO: `/v3/users/ad7ecfe96cd944a29861ce8f66fc6eb0` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:30.488782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad7ecfe96cd944a29861ce8f66fc6eb0)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:30.491023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:30.718549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad7ecfe96cd944a29861ce8f66fc6eb0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:30.719549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad7ecfe96cd944a29861ce8f66fc6eb0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:30.739688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad7ecfe96cd944a29861ce8f66fc6eb0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:30.749114 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad7ecfe96cd944a29861ce8f66fc6eb0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:30.750122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bce4b919-8530-4c4d-8997-96845dadefec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad7ecfe96cd944a29861ce8f66fc6eb0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:30.817302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d3cde8-2586-44e7-b713-eb104cfb7246 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:30.817903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d3cde8-2586-44e7-b713-eb104cfb7246 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:30.817903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d3cde8-2586-44e7-b713-eb104cfb7246 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:30.819257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 669/1323] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:05:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:05:30.833209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 655/1324] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:30 2022] DELETE /identity/v3/users/ad7ecfe96cd944a29861ce8f66fc6eb0 => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:30.836658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:30.837143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:30.837623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:30.850078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:30.850777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:30.851446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:30.888281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:30.890351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:30.907272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:30.908616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:30.927711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:31.033640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 656/1325] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:05:30 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:31.056637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 670/1326] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:31.106043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:31.108337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:31.133138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb70d16-2461-4131-8e2a-a2283fb5182c None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:31.149546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:31.150478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:31.154075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:31.154705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:31.165558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:31.167300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:31.185505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:31.189079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:31.224700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 657/1327] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:31.288151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f314566-819e-4ecc-a308-41f85a85ac05 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:31.296278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708407ec-59b9-47c8-a0a7-af80d49fb82f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:31.300066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708407ec-59b9-47c8-a0a7-af80d49fb82f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:31.300548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708407ec-59b9-47c8-a0a7-af80d49fb82f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:31.300978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708407ec-59b9-47c8-a0a7-af80d49fb82f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:31.321045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708407ec-59b9-47c8-a0a7-af80d49fb82f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:31.322530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708407ec-59b9-47c8-a0a7-af80d49fb82f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:31.367778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 671/1328] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:05:32.108151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:32.111155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:32.140857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87670edd-1632-4936-834c-dc6d276780ff service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:32.153473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:32.155507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:32.156258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:32.156793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] PATH_INFO: `/v3/users/f9eaff8ed7204e6e8fddbd983f1d20ee` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:32.158315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9eaff8ed7204e6e8fddbd983f1d20ee)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:32.162028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:32.530407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9eaff8ed7204e6e8fddbd983f1d20ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:32.541551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9eaff8ed7204e6e8fddbd983f1d20ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:32.587453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9eaff8ed7204e6e8fddbd983f1d20ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:32.610985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9eaff8ed7204e6e8fddbd983f1d20ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:32.612640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f25ed7d8-050d-481f-943d-13f1779bc6dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9eaff8ed7204e6e8fddbd983f1d20ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:32.775948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 658/1329] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:32 2022] DELETE /identity/v3/users/f9eaff8ed7204e6e8fddbd983f1d20ee => generated 0 bytes in 673 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:32.780589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:32.783262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:32.783262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:32.815467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:32.816731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:32.862809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:32.863092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:32.863727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:32.907034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:32.919196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] MFA Rules not processed for user `724baaa4aa834e609fe4f513f9fc0b4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:32.920704 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:32.930332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 672/1330] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:32.941723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:32.943667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:33.050496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 659/1331] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:33.050726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e8359d-5707-492f-b5c5-866d075ff51e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:33.053366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:33.060930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:33.060930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:33.060930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:33.061619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:33.061715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:33.069698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:33.070181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:33.072590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:33.085791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 673/1332] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:33.249406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98225ffb-d1d5-4209-9377-193ad538ff4f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:33.274287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:33.276357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:33.280159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:33.280159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:33.305699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:33.308296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:33.321593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 660/1333] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:33.674188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:33.675693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:33.689659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a187182-b326-4255-8d43-61eaafc6458c service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:33.694900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:33.695539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:33.695823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:33.696101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] PATH_INFO: `/v3/projects/bb6cdda7411a4d16880f5f52725afdec` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:33.697343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb6cdda7411a4d16880f5f52725afdec)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:33.699300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:33.860402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb6cdda7411a4d16880f5f52725afdec', 'request_id': 'req-49684b9b-65f1-4852-be90-23ffa13e5bde'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:33.990872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb6cdda7411a4d16880f5f52725afdec', 'request_id': 'req-49684b9b-65f1-4852-be90-23ffa13e5bde'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.010775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb6cdda7411a4d16880f5f52725afdec', 'request_id': 'req-49684b9b-65f1-4852-be90-23ffa13e5bde'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.058340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] The token cache is being invalidate because project bb6cdda7411a4d16880f5f52725afdec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:34.058761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.063432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.065647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 674/1334] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:33 2022] DELETE /identity/v3/projects/bb6cdda7411a4d16880f5f52725afdec => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:34.070823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:34.071841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:34.073822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:34.099264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553fe204-4a84-47bb-a34d-9a93a320470c service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:34.272321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:34.273807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:34.274087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:34.274369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] PATH_INFO: `/v3/users/2e482f31f548492ead2875091dbc1e70` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:34.280928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e482f31f548492ead2875091dbc1e70)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:34.286798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:34.293914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49684b9b-65f1-4852-be90-23ffa13e5bde admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:34.306543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:34.307248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:34.307592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:34.307913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] PATH_INFO: `/v3/projects/f6c2f7094bcf43c49ea03ea775595f51` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:34.310916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6c2f7094bcf43c49ea03ea775595f51)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:34.314918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:34.457740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:34.528248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e482f31f548492ead2875091dbc1e70'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.533983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e482f31f548492ead2875091dbc1e70'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.546351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6c2f7094bcf43c49ea03ea775595f51', 'request_id': 'req-4d22ac5d-3714-427d-b90c-097ca2084781'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.586201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e482f31f548492ead2875091dbc1e70'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.604513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e482f31f548492ead2875091dbc1e70'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.607982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-af03e760-9946-4a98-8abe-7137293c1aa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e482f31f548492ead2875091dbc1e70'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.630034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6c2f7094bcf43c49ea03ea775595f51', 'request_id': 'req-4d22ac5d-3714-427d-b90c-097ca2084781'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.639399 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6c2f7094bcf43c49ea03ea775595f51', 'request_id': 'req-4d22ac5d-3714-427d-b90c-097ca2084781'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.662560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] The token cache is being invalidate because project f6c2f7094bcf43c49ea03ea775595f51 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:34.662864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.667648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:34.679522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 675/1335] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:34 2022] DELETE /identity/v3/projects/f6c2f7094bcf43c49ea03ea775595f51 => generated 0 bytes in 603 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:34.680882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:34.691038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:34.737226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 661/1336] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:34 2022] DELETE /identity/v3/users/2e482f31f548492ead2875091dbc1e70 => generated 0 bytes in 668 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:34.744235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a945761-23a3-4fdb-a81c-669cfc8a9e5a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:34.744531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a945761-23a3-4fdb-a81c-669cfc8a9e5a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:34.746819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a945761-23a3-4fdb-a81c-669cfc8a9e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:34.794514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:34.806681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3a945761-23a3-4fdb-a81c-669cfc8a9e5a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:34.808194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3a945761-23a3-4fdb-a81c-669cfc8a9e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:34.881896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22ac5d-3714-427d-b90c-097ca2084781 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:35.025167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 662/1337] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:35.033100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218a38cd-31fe-43b8-bee2-bd5ceef9c2b8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:35.033425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218a38cd-31fe-43b8-bee2-bd5ceef9c2b8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:35.033712 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218a38cd-31fe-43b8-bee2-bd5ceef9c2b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:35.052263 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:35.052945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:35.053234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:35.058631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] PATH_INFO: `/v3/users/c7931041aa5a4546bfeaf5e8cfe50f41` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:35.058631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7931041aa5a4546bfeaf5e8cfe50f41)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:35.058631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:05:35.058631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Oct 28 20:05:35.058631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: Oct 28 20:05:35.067469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:35.104483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-218a38cd-31fe-43b8-bee2-bd5ceef9c2b8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:35.105775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-218a38cd-31fe-43b8-bee2-bd5ceef9c2b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:35.143974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 663/1338] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:35.144924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218a38cd-31fe-43b8-bee2-bd5ceef9c2b8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:35.189091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218a38cd-31fe-43b8-bee2-bd5ceef9c2b8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:35.207622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:35.208201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:35.208492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:35.208772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:35.209479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:35.209804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:35.211486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:35.250568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 664/1339] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:35.252466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:35.286809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8266db92-55f5-461f-a558-47a39ca3854f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:35.306506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:35.306989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:35.307828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:35.308117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:35.308777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:35.324371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:35.379810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7931041aa5a4546bfeaf5e8cfe50f41'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:35.449226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 665/1340] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:35 2022] POST /identity/v3/projects => generated 369 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:35.451937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:35.480115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7931041aa5a4546bfeaf5e8cfe50f41'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:35.551317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7931041aa5a4546bfeaf5e8cfe50f41'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:35.552016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7931041aa5a4546bfeaf5e8cfe50f41'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:35.552514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7931041aa5a4546bfeaf5e8cfe50f41'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:35.586035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55858a7-c97c-4f58-82f9-1fe3ba0e8141 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:35.588002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 676/1341] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:34 2022] DELETE /identity/v3/users/c7931041aa5a4546bfeaf5e8cfe50f41 => generated 0 bytes in 908 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:35.675934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:35.719106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:35.720031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:35.720496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:35.720935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:35.724110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:35.728468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:35.865728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 666/1342] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:35 2022] POST /identity/v3/users => generated 285 bytes in 415 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:05:35.866036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:35.867967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:35.905753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefe210d-6ca5-476e-8448-72045951cab2 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:35.926821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:35.930611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:35.930611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:35.930611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:35.930611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:35.932113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:35.949542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 677/1343] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:35 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:05:35.951115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:35.956925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:35.991230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8021c209-5703-48bb-9b0d-ff8b26beafd8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:36.004931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a065dc22-a0ec-48a0-b631-d7a07eed9887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:36.007088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065dc22-a0ec-48a0-b631-d7a07eed9887 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:36.007088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065dc22-a0ec-48a0-b631-d7a07eed9887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:36.007088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065dc22-a0ec-48a0-b631-d7a07eed9887 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:36.008146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a065dc22-a0ec-48a0-b631-d7a07eed9887 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:36.012326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a065dc22-a0ec-48a0-b631-d7a07eed9887 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:36.047206 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:36.057398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 667/1344] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:35 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:36.064052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab39f3c7-6a16-444c-af31-41ad066edbfc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:36.071471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:36.072207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:36.072573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:36.072987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] PATH_INFO: `/v3/projects/cd9d8e3aaecc4f0e9659b1c730f1aa7a/users/a938c8acd4134f2dab23bc8c5dfc8d1f/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:36.100926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd9d8e3aaecc4f0e9659b1c730f1aa7a, user_id=a938c8acd4134f2dab23bc8c5dfc8d1f, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:36.103863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:36.210064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2352c5d7-ea9e-48d1-ad7f-c584b394c0f2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:36.210613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2352c5d7-ea9e-48d1-ad7f-c584b394c0f2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:36.211949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2352c5d7-ea9e-48d1-ad7f-c584b394c0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:36.219415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 678/1345] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:36 2022] PUT /identity/v3/projects/cd9d8e3aaecc4f0e9659b1c730f1aa7a/users/a938c8acd4134f2dab23bc8c5dfc8d1f/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:36.266746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2352c5d7-ea9e-48d1-ad7f-c584b394c0f2 None None] MFA Rules not processed for user `a938c8acd4134f2dab23bc8c5dfc8d1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:36.270944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2352c5d7-ea9e-48d1-ad7f-c584b394c0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:36.379167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 668/1346] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:36 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:36.381650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:36.500729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e159f36-f74f-4d69-ab83-06ac2d7af6ee admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:36.507205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4083483e-46c4-401a-bcd5-3ae7afe2c02f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:36.507976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4083483e-46c4-401a-bcd5-3ae7afe2c02f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:36.508336 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4083483e-46c4-401a-bcd5-3ae7afe2c02f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:36.509302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4083483e-46c4-401a-bcd5-3ae7afe2c02f admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:36.510053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4083483e-46c4-401a-bcd5-3ae7afe2c02f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:36.512394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4083483e-46c4-401a-bcd5-3ae7afe2c02f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:36.608309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 679/1347] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:36 2022] POST /identity/v3/projects => generated 367 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:36.608874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2352c5d7-ea9e-48d1-ad7f-c584b394c0f2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:36.624189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2352c5d7-ea9e-48d1-ad7f-c584b394c0f2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:36.715602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:36.716659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:36.717141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:36.755680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:36.760477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:36.760477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:36.760477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:36.765272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:36.767887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:36.811138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] MFA Rules not processed for user `15db5106efbd434f99a4d102810cc7a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:36.813714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:36.910226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 669/1348] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:36 2022] POST /identity/v3/users => generated 290 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:05:36.911268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:36.921747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:36.954525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb6e6c-f6a0-4d7e-9c5c-48b6439ba983 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:36.968837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:36.969527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:36.972755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:36.973749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] PATH_INFO: `/v3/projects/c85093aea4804fb6bf7e9de10f17e434` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:36.977625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c85093aea4804fb6bf7e9de10f17e434)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:36.983274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: WARNING py.warnings [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 20:05:36.983274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: warnings.warn(msg) Oct 28 20:05:36.983274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: Oct 28 20:05:36.987078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:37.030049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 680/1349] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:37.032068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:37.037789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:37.064005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41763ac1-64a1-40b4-b539-598654cb38db None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:37.085655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:37.087634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:37.088284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:37.090644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] PATH_INFO: `/v3/projects/51a485932b6b4e0086066885364eb974` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:37.094781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] RBAC: Authorizing `identity:delete_project(project_id=51a485932b6b4e0086066885364eb974)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:37.097439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:37.119915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:INSERT=3,keystone:DELETE=1,keystone:SELECT=96 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:37.192080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c85093aea4804fb6bf7e9de10f17e434', 'request_id': 'req-82accd21-7d14-46f5-a040-f749cd4afa45'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.216220 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c85093aea4804fb6bf7e9de10f17e434', 'request_id': 'req-82accd21-7d14-46f5-a040-f749cd4afa45'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.234982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51a485932b6b4e0086066885364eb974', 'request_id': 'req-76544bea-3330-4d4e-a381-112e3ca1295f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.251695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c85093aea4804fb6bf7e9de10f17e434', 'request_id': 'req-82accd21-7d14-46f5-a040-f749cd4afa45'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.254840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:37.301028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51a485932b6b4e0086066885364eb974', 'request_id': 'req-76544bea-3330-4d4e-a381-112e3ca1295f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.361486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51a485932b6b4e0086066885364eb974', 'request_id': 'req-76544bea-3330-4d4e-a381-112e3ca1295f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.676749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] The token cache is being invalidate because project 51a485932b6b4e0086066885364eb974 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:37.676749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.676749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.676749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 681/1350] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:37 2022] DELETE /identity/v3/projects/51a485932b6b4e0086066885364eb974 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:37.676749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:37.678650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:37.681955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] The token cache is being invalidate because project c85093aea4804fb6bf7e9de10f17e434 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:37.681955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.681955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:37.681955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 670/1351] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:36 2022] DELETE /identity/v3/projects/c85093aea4804fb6bf7e9de10f17e434 => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:37.681955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:37.683763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:37.684586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76544bea-3330-4d4e-a381-112e3ca1295f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:37.684586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:37.684586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:37.684586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:37.686125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:37.686125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:37.686125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:37.687145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82accd21-7d14-46f5-a040-f749cd4afa45 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:37.699686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 682/1352] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:37 2022] GET /identity/v3/roles => generated 1420 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:37.709010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:37.712790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:37.751551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6577bd72-aec4-4781-9697-f40b2de52763 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:37.767053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:37.767714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:37.767997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:37.768418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] PATH_INFO: `/v3/users/ca20502cea544288803af97a4da45d8c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:37.772235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca20502cea544288803af97a4da45d8c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:37.774559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:37.890071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93eb29d2-ae13-4b0e-90ff-39704988c315 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:37.890727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eb29d2-ae13-4b0e-90ff-39704988c315 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:37.892432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eb29d2-ae13-4b0e-90ff-39704988c315 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:37.895946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eb29d2-ae13-4b0e-90ff-39704988c315 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:37.900724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-93eb29d2-ae13-4b0e-90ff-39704988c315 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:38.033228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93eb29d2-ae13-4b0e-90ff-39704988c315 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:38.036056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93eb29d2-ae13-4b0e-90ff-39704988c315 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:38.049053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 671/1353] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:37 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 608 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:38.050832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.051367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.051743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.071142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca20502cea544288803af97a4da45d8c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:38.114244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:38.117018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:38.124289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca20502cea544288803af97a4da45d8c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:38.133207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 672/1354] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:38.133207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:38.133207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:38.150003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d929a522-e994-4f86-b84e-6a1b450bbbe9 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:38.157233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:38.159212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.159534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.159935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] PATH_INFO: `/v3/projects/1e6622c4b6fe4dbda72b6b9702b32e3f/users/1d213c0c12314059a31d6185d3136cab/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.187506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e6622c4b6fe4dbda72b6b9702b32e3f, user_id=1d213c0c12314059a31d6185d3136cab, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:38.191837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:38.224546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca20502cea544288803af97a4da45d8c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:38.225450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca20502cea544288803af97a4da45d8c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:38.226428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cd7943a1-c0b6-4276-9310-d2d65e9e1ef7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca20502cea544288803af97a4da45d8c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:38.254627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 683/1355] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:37 2022] DELETE /identity/v3/users/ca20502cea544288803af97a4da45d8c => generated 0 bytes in 554 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:38.264440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5997e7-0170-49f4-bafb-c7bdb8506464 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.265660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5997e7-0170-49f4-bafb-c7bdb8506464 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.266010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5997e7-0170-49f4-bafb-c7bdb8506464 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.290659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 673/1356] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:38 2022] PUT /identity/v3/projects/1e6622c4b6fe4dbda72b6b9702b32e3f/users/1d213c0c12314059a31d6185d3136cab/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:38.302815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:38.306848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:38.352213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ba5997e7-0170-49f4-bafb-c7bdb8506464 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:38.355932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ba5997e7-0170-49f4-bafb-c7bdb8506464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:38.500110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f46d09-604c-4a6b-a58b-0ab60457443e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:38.510145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:38.511510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.512194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.512931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.514069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:38.517457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:38.520068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 684/1357] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:38.525767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5997e7-0170-49f4-bafb-c7bdb8506464 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:38.542518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5997e7-0170-49f4-bafb-c7bdb8506464 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:38.545680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 674/1358] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:38 2022] GET /identity/v3/roles => generated 1420 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:38.555183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:38.555318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:38.555318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.555560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.555851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.556575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:38.556896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:38.564699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:38.570293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:38.589891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df76d631-bfdd-4e51-a79b-171653ef0c71 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:38.594053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 685/1359] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:05:38.603028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:38.610598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868e4ed8-d857-4c51-a640-5b27db644e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:38.612410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868e4ed8-d857-4c51-a640-5b27db644e8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.612410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868e4ed8-d857-4c51-a640-5b27db644e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.612410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868e4ed8-d857-4c51-a640-5b27db644e8f admin admin] PATH_INFO: `/v3/projects/1e6622c4b6fe4dbda72b6b9702b32e3f/users/1d213c0c12314059a31d6185d3136cab/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.616421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868e4ed8-d857-4c51-a640-5b27db644e8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e6622c4b6fe4dbda72b6b9702b32e3f, user_id=1d213c0c12314059a31d6185d3136cab, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:38.618743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868e4ed8-d857-4c51-a640-5b27db644e8f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:38.654413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae7cf3-e29b-4e6b-8533-1b0651c101ca admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:38.658826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 675/1360] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:38 2022] PUT /identity/v3/projects/1e6622c4b6fe4dbda72b6b9702b32e3f/users/1d213c0c12314059a31d6185d3136cab/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:38.664661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.667834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.668226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.754383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] MFA Rules not processed for user `1d213c0c12314059a31d6185d3136cab`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:38.758863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:38.804013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:38.804623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:38.804915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:38.805363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:38.805875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:38.807833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:38.932192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 686/1361] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:38 2022] POST /identity/v3/projects => generated 385 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:38.937149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:39.034947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=1,keystone:SELECT=98,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:39.040156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 676/1362] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:38 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 377 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:39.048681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:39.050865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:39.082502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dad5ab-28e6-43de-9cc5-53ea59cd472f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:39.094856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:39.100436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:39.100436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:39.100436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:39.100436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:39.103568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:39.220482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cba0d4e-9513-4032-a505-eded9fa45e0c None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:39.258866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 687/1363] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:38 2022] POST /identity/v3/users => generated 293 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:39.262423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:39.287900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77847f72-28f3-42bb-9523-065768c4d153 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:39.314669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:39.315491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:39.315879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:39.316412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:39.317172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:39.318966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:39.360479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 688/1364] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:39 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:39.364719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:39.378140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1638423-1ace-4b04-a9bd-f19957e0de4c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:39.381101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1638423-1ace-4b04-a9bd-f19957e0de4c service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:39.382443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1638423-1ace-4b04-a9bd-f19957e0de4c service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:39.382976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1638423-1ace-4b04-a9bd-f19957e0de4c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:39.385354 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d90fab7-2fa7-4965-8c05-4c715f232c66 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:39.400429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:39.401375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:39.401803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:39.402282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:39.408205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:39.408205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:39.442898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 689/1365] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:39 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:39.445279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:39.479623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0694e7e4-3231-4394-8f9a-ea0cd6a8a7f6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:39.496927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:39.496927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:39.499612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:39.499922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] PATH_INFO: `/v3/projects/3a4347346ddd45eaa9b05e1afd15c194/users/37cbb375d7eb4b0489d14ea7c5c26a9c/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:39.542371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a4347346ddd45eaa9b05e1afd15c194, user_id=37cbb375d7eb4b0489d14ea7c5c26a9c, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:39.546689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:39.547482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1638423-1ace-4b04-a9bd-f19957e0de4c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:39.549067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1638423-1ace-4b04-a9bd-f19957e0de4c service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:39.573852 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 677/1366] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:39 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 527 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:05:39.657629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 690/1367] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:39 2022] PUT /identity/v3/projects/3a4347346ddd45eaa9b05e1afd15c194/users/37cbb375d7eb4b0489d14ea7c5c26a9c/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:39.669085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01821454-7983-4e36-9f12-816e930afea5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:39.670258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01821454-7983-4e36-9f12-816e930afea5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:39.670792 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01821454-7983-4e36-9f12-816e930afea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:39.746611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:39.748872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:39.762157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-01821454-7983-4e36-9f12-816e930afea5 None None] MFA Rules not processed for user `37cbb375d7eb4b0489d14ea7c5c26a9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:39.770412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-01821454-7983-4e36-9f12-816e930afea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:39.835604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:39.930826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb44c6c-bb8f-4347-8d47-f09c9fbbefe3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:39.939970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da64c8-5a49-4a07-95fa-a9b560b4cfa3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:39.941999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da64c8-5a49-4a07-95fa-a9b560b4cfa3 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:39.942469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da64c8-5a49-4a07-95fa-a9b560b4cfa3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:39.943630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da64c8-5a49-4a07-95fa-a9b560b4cfa3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:39.947462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d5da64c8-5a49-4a07-95fa-a9b560b4cfa3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:39.956685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:39.980206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 678/1368] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:39.991343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01821454-7983-4e36-9f12-816e930afea5 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:40.018615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01821454-7983-4e36-9f12-816e930afea5 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:40.039931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:40.040707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:40.041086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:40.041489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:40.060639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:40.063381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:40.088391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 679/1369] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:05:40.112506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5da64c8-5a49-4a07-95fa-a9b560b4cfa3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:40.116111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5da64c8-5a49-4a07-95fa-a9b560b4cfa3 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:40.122612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:40.147416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 691/1370] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:39 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:40.284588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a087a076-9c07-4b4d-b3a0-3d1fa597846c service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:40.285648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:40.293163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:40.296072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:40.296497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:40.297083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] PATH_INFO: `/v3/users/37cbb375d7eb4b0489d14ea7c5c26a9c` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:40.299541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] RBAC: Authorizing `identity:delete_user(user_id=37cbb375d7eb4b0489d14ea7c5c26a9c)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:40.302188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:40.381001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:40.381987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:40.382398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:40.439000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] MFA Rules not processed for user `e0ba6bf2459d4a98a3a1f5684bdad2b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:40.440429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:40.591830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 692/1371] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:40 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:40.607306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:40.609547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:40.643160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37cbb375d7eb4b0489d14ea7c5c26a9c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:40.643867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37cbb375d7eb4b0489d14ea7c5c26a9c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:40.644540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:40.708479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37cbb375d7eb4b0489d14ea7c5c26a9c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:40.744951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37cbb375d7eb4b0489d14ea7c5c26a9c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:40.747785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37cbb375d7eb4b0489d14ea7c5c26a9c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:40.794352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca0fd7-da1c-4bcc-a927-13159f67be33 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:40.801703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eee6413-402c-4904-9c3d-45493939750c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:40.802405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eee6413-402c-4904-9c3d-45493939750c service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:40.802721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eee6413-402c-4904-9c3d-45493939750c service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:40.803000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eee6413-402c-4904-9c3d-45493939750c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:40.804943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7eee6413-402c-4904-9c3d-45493939750c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:40.826981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eee6413-402c-4904-9c3d-45493939750c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:40.830527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eee6413-402c-4904-9c3d-45493939750c service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:40.836342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 680/1372] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:40 2022] DELETE /identity/v3/users/37cbb375d7eb4b0489d14ea7c5c26a9c => generated 0 bytes in 714 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:40.840398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:40.864492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:40.865852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 693/1373] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:40 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:40.869679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98c399d-01dc-4812-af46-94bf3b7799ed None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:40.872105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98c399d-01dc-4812-af46-94bf3b7799ed None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:40.876511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98c399d-01dc-4812-af46-94bf3b7799ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:40.925529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a98c399d-01dc-4812-af46-94bf3b7799ed None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:40.929065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a98c399d-01dc-4812-af46-94bf3b7799ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:41.074282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18107ad2-9371-4055-8e9a-1ce2e439790e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:41.100316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:41.102660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:41.102947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:41.105464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] PATH_INFO: `/v3/projects/79ab60d1d83e4e59bc92904965ccbf37` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:41.126639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79ab60d1d83e4e59bc92904965ccbf37)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:41.131806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:41.146548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 694/1374] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:41.194539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:41.195020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:41.195496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:41.291705 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] MFA Rules not processed for user `ce7f52b7f96c400fb7d5fb0453c94362`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:41.298323 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:41.454253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79ab60d1d83e4e59bc92904965ccbf37', 'request_id': 'req-d57ec3d6-1710-4533-9f20-6c6ae3a98554'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:41.478249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79ab60d1d83e4e59bc92904965ccbf37', 'request_id': 'req-d57ec3d6-1710-4533-9f20-6c6ae3a98554'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:41.503383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79ab60d1d83e4e59bc92904965ccbf37', 'request_id': 'req-d57ec3d6-1710-4533-9f20-6c6ae3a98554'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:41.586856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:41.592739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] The token cache is being invalidate because project 79ab60d1d83e4e59bc92904965ccbf37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:41.593047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:41.594379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d57ec3d6-1710-4533-9f20-6c6ae3a98554 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:41.596428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 695/1375] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:41 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 403 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:41.597664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:41.598136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 681/1376] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:40 2022] DELETE /identity/v3/projects/79ab60d1d83e4e59bc92904965ccbf37 => generated 0 bytes in 761 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:41.599447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:41.610628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e612d2-9087-479f-96f8-22fd9dee5ef3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:41.611205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e612d2-9087-479f-96f8-22fd9dee5ef3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:41.612189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e612d2-9087-479f-96f8-22fd9dee5ef3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:41.676523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f2e612d2-9087-479f-96f8-22fd9dee5ef3 None None] MFA Rules not processed for user `ce7f52b7f96c400fb7d5fb0453c94362`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:41.683635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f2e612d2-9087-479f-96f8-22fd9dee5ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:41.699418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:41.763755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db916775-85b3-455a-b13e-5e7e3b7a6916 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:41.811300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 682/1377] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:41 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:41.813130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:41.813594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:41.814059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:41.857548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:41.861262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:41.893298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47386341-f217-42ee-af05-fe872e8e8841 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:41.894552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47386341-f217-42ee-af05-fe872e8e8841 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:41.894847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47386341-f217-42ee-af05-fe872e8e8841 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:41.896575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47386341-f217-42ee-af05-fe872e8e8841 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:41.900872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-47386341-f217-42ee-af05-fe872e8e8841 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:41.989368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 683/1378] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:41.990428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:41.992586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:42.086699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47386341-f217-42ee-af05-fe872e8e8841 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:42.089203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47386341-f217-42ee-af05-fe872e8e8841 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:42.115277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 696/1379] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:42.120854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01049468-4608-4d62-ad31-4f8e9f8f2b3b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:42.120967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01049468-4608-4d62-ad31-4f8e9f8f2b3b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:42.121240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01049468-4608-4d62-ad31-4f8e9f8f2b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:42.162349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-01049468-4608-4d62-ad31-4f8e9f8f2b3b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:42.167420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-01049468-4608-4d62-ad31-4f8e9f8f2b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:42.188726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 697/1380] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:42.190520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b4e27a-8447-4af4-a32b-2e38181d6476 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:42.191285 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b4e27a-8447-4af4-a32b-2e38181d6476 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:42.191742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b4e27a-8447-4af4-a32b-2e38181d6476 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:42.207993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e6843b-2206-43ba-831b-09169a48b668 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:42.215383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:42.216784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:42.217497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:42.219021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:42.220027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:42.239423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:42.244659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:42.254341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-41b4e27a-8447-4af4-a32b-2e38181d6476 None None] MFA Rules not processed for user `724baaa4aa834e609fe4f513f9fc0b4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:42.255911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-41b4e27a-8447-4af4-a32b-2e38181d6476 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:42.272188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 684/1381] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:41 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:42.272188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:42.289821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f08468-75c7-49d5-9ebb-7bf8f2613cd9 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:42.305285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:42.306341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:42.306786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:42.307215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:42.307883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:42.308226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:42.310421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:42.345313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 685/1382] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:42.346308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:42.380789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e896a-8a5f-4117-afed-43df763b8362 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:42.400401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 698/1383] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:42.405295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:42.406145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:42.406576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:42.409270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:42.410130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:42.412085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:42.510204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:42.566539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 686/1384] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:42 2022] POST /identity/v3/projects => generated 373 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:42.571378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b4e27a-8447-4af4-a32b-2e38181d6476 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:42.596089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b4e27a-8447-4af4-a32b-2e38181d6476 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:42.710911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:42.785191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:42.786050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:42.786450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:42.786831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:42.787599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:42.789765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:42.920297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 699/1385] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:42 2022] POST /identity/v3/users => generated 287 bytes in 350 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:42.920944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:42.939456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec329fb-4d6c-436b-b5d7-b881ef6f8742 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:42.948753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:42.949454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:42.949751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:42.950026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:42.950998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:42.952794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:42.985689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 687/1386] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:42 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:42.989240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:43.005236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534af27-d5d4-4b99-bdaf-1736d5510b45 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:43.035308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202347e0-5e56-48ff-a303-b693ccddeeff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:43.036562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202347e0-5e56-48ff-a303-b693ccddeeff admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.036968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202347e0-5e56-48ff-a303-b693ccddeeff admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.037381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202347e0-5e56-48ff-a303-b693ccddeeff admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.038313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202347e0-5e56-48ff-a303-b693ccddeeff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:43.043245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202347e0-5e56-48ff-a303-b693ccddeeff admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:43.073788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 700/1387] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:42 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:43.084418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:43.104089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cab45ed-1997-42c2-8b56-7ab5ff2fc4ce admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:43.115264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d3ce96-c7fd-4791-83e9-65971f058287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:43.115956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d3ce96-c7fd-4791-83e9-65971f058287 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.116239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d3ce96-c7fd-4791-83e9-65971f058287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.116516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d3ce96-c7fd-4791-83e9-65971f058287 admin admin] PATH_INFO: `/v3/projects/aa35fbee77ab41d1a2db41f5f8bb39bf/users/2a21b8fb2dc54513bd5256eb05b19055/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.154517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d3ce96-c7fd-4791-83e9-65971f058287 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa35fbee77ab41d1a2db41f5f8bb39bf, user_id=2a21b8fb2dc54513bd5256eb05b19055, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:43.160273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d3ce96-c7fd-4791-83e9-65971f058287 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:43.294986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 688/1388] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:43 2022] PUT /identity/v3/projects/aa35fbee77ab41d1a2db41f5f8bb39bf/users/2a21b8fb2dc54513bd5256eb05b19055/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:43.305432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b604d0a7-b7cb-4f22-b01a-4b4883fec0fe None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.307661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b604d0a7-b7cb-4f22-b01a-4b4883fec0fe None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.308037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b604d0a7-b7cb-4f22-b01a-4b4883fec0fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.381162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b604d0a7-b7cb-4f22-b01a-4b4883fec0fe None None] MFA Rules not processed for user `2a21b8fb2dc54513bd5256eb05b19055`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:43.384095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b604d0a7-b7cb-4f22-b01a-4b4883fec0fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:43.479076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5101059-4c19-4af3-a71a-59846850929f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.479621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5101059-4c19-4af3-a71a-59846850929f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.479965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5101059-4c19-4af3-a71a-59846850929f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.548084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d5101059-4c19-4af3-a71a-59846850929f None None] MFA Rules not processed for user `1b3396bc67ae4f6c9c8808653fef4676`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:43.551112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d5101059-4c19-4af3-a71a-59846850929f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:43.574827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 701/1389] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:43.577751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b604d0a7-b7cb-4f22-b01a-4b4883fec0fe None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:43.734957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 689/1390] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:43.743551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2d0a9a-ca23-4ee7-9ce8-09b69067cdfd None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.744370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2d0a9a-ca23-4ee7-9ce8-09b69067cdfd None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.747107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2d0a9a-ca23-4ee7-9ce8-09b69067cdfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.809577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6e2d0a9a-ca23-4ee7-9ce8-09b69067cdfd None None] MFA Rules not processed for user `1b3396bc67ae4f6c9c8808653fef4676`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:43.813760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6e2d0a9a-ca23-4ee7-9ce8-09b69067cdfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:43.836949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b604d0a7-b7cb-4f22-b01a-4b4883fec0fe None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:43.846142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:43.847056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.847451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.847808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.848647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 690/1391] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:43.851067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85874a7a-dd8f-45a3-baa4-592763f8db6e None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.853287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:43.855462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:43.855953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85874a7a-dd8f-45a3-baa4-592763f8db6e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.856393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85874a7a-dd8f-45a3-baa4-592763f8db6e None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.859279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 691/1392] 10.222.0.66 () {54 vars in 912 bytes} [Fri Oct 28 20:05:43 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:05:43.872667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:43.876696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:43.881496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:43.942246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] MFA Rules not processed for user `91639dc527ae47e892164544572cbec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:43.944187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:43.996696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 702/1393] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:43 2022] POST /identity/v3/projects => generated 369 bytes in 421 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:44.000243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.083440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:44.101662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 692/1394] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:05:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:44.132922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af27bcf8-2b88-47b9-baca-c52ead9c352c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:44.133629 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.135299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:44.141682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:44.142333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.142685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.143044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.145792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:44.152085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:44.156789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459073ce-45bf-42aa-be6e-ac9517c2e7d9 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:44.239778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:44.259618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 703/1395] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:43 2022] POST /identity/v3/users => generated 291 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:44.264185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.265181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:44.265882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.266167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.266498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.282452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:44.284018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:44.290281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122a8a3b-a97d-4eca-9f5c-a922b1fa9d52 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:44.296852 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 693/1396] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:44.299826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:44.300483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.300771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.301050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.301802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:44.303486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:44.318711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 704/1397] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:44 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:44.320416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.334004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df92ccf4-c673-448f-8683-911cb25a8a37 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:44.339516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:44.340214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.340502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.340784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] PATH_INFO: `/v3/projects/69ee9bc5872241aa90e08f04008add89/users/26b4805f8d934619ac02ab08b83ab6a4/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.363042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] RBAC: Authorizing `identity:create_grant(project_id=69ee9bc5872241aa90e08f04008add89, user_id=26b4805f8d934619ac02ab08b83ab6a4, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:44.367636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:44.401275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.404868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:44.437258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786d3756-2758-43ea-987b-c3ebd1a42c10 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:44.446669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 694/1398] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:44 2022] PUT /identity/v3/projects/69ee9bc5872241aa90e08f04008add89/users/26b4805f8d934619ac02ab08b83ab6a4/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:44.448078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.593017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:44.593627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.593907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.594182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] PATH_INFO: `/v3/projects/3a4347346ddd45eaa9b05e1afd15c194` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.595302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a4347346ddd45eaa9b05e1afd15c194)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:44.597259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:44.602614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e48a3-75fe-4cb0-9d25-48b1cb05320b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:44.608179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6841464f-ef9d-4011-b474-6342f4a63552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:44.609297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841464f-ef9d-4011-b474-6342f4a63552 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.609781 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841464f-ef9d-4011-b474-6342f4a63552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.610263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841464f-ef9d-4011-b474-6342f4a63552 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.611018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6841464f-ef9d-4011-b474-6342f4a63552 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:44.612730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6841464f-ef9d-4011-b474-6342f4a63552 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:44.626333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 695/1399] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:44 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:05:44.628281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8597bdc4-2c71-4499-a05d-9aa3ddf8a886 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.628706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8597bdc4-2c71-4499-a05d-9aa3ddf8a886 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.628995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8597bdc4-2c71-4499-a05d-9aa3ddf8a886 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.630488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 696/1400] 10.222.0.66 () {54 vars in 912 bytes} [Fri Oct 28 20:05:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:05:44.637495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8597bdc4-2c71-4499-a05d-9aa3ddf8a886 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.645364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8597bdc4-2c71-4499-a05d-9aa3ddf8a886 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:44.650392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8a6b7-7d68-41f3-8eda-2c2456483e37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:44.651310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8a6b7-7d68-41f3-8eda-2c2456483e37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.651594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8a6b7-7d68-41f3-8eda-2c2456483e37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.651866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8a6b7-7d68-41f3-8eda-2c2456483e37 admin admin] PATH_INFO: `/v3/projects/69ee9bc5872241aa90e08f04008add89/users/26b4805f8d934619ac02ab08b83ab6a4/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.654431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc8a6b7-7d68-41f3-8eda-2c2456483e37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69ee9bc5872241aa90e08f04008add89, user_id=26b4805f8d934619ac02ab08b83ab6a4, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:44.657308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc8a6b7-7d68-41f3-8eda-2c2456483e37 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:44.673422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 697/1401] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:44 2022] PUT /identity/v3/projects/69ee9bc5872241aa90e08f04008add89/users/26b4805f8d934619ac02ab08b83ab6a4/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:44.677528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8301f-7448-4fe1-9afd-06472c9e6203 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.677931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8301f-7448-4fe1-9afd-06472c9e6203 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.678693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8301f-7448-4fe1-9afd-06472c9e6203 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.703871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-74d8301f-7448-4fe1-9afd-06472c9e6203 None None] MFA Rules not processed for user `91639dc527ae47e892164544572cbec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:44.705743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-74d8301f-7448-4fe1-9afd-06472c9e6203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:44.770109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 698/1402] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:05:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:44.771901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff2fbf6-ba2e-4845-9dcb-55479c8b18a8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.772384 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff2fbf6-ba2e-4845-9dcb-55479c8b18a8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.772868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff2fbf6-ba2e-4845-9dcb-55479c8b18a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:44.810379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a4347346ddd45eaa9b05e1afd15c194', 'request_id': 'req-005456c0-4b3b-470a-bf24-f2f5c9e20368'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:44.824089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-aff2fbf6-ba2e-4845-9dcb-55479c8b18a8 None None] MFA Rules not processed for user `26b4805f8d934619ac02ab08b83ab6a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:44.825696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-aff2fbf6-ba2e-4845-9dcb-55479c8b18a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:44.899243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a4347346ddd45eaa9b05e1afd15c194', 'request_id': 'req-005456c0-4b3b-470a-bf24-f2f5c9e20368'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:44.913995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a4347346ddd45eaa9b05e1afd15c194', 'request_id': 'req-005456c0-4b3b-470a-bf24-f2f5c9e20368'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:44.942424 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] The token cache is being invalidate because project 3a4347346ddd45eaa9b05e1afd15c194 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:44.942737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:44.943697 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:44.953455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 705/1403] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:44 2022] DELETE /identity/v3/projects/3a4347346ddd45eaa9b05e1afd15c194 => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:44.955612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:44.958795 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:44.978306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 699/1404] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:44 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:44.981133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50589c6e-1307-4f8c-9c4e-9ef76586d99e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:44.981760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50589c6e-1307-4f8c-9c4e-9ef76586d99e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:44.982303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50589c6e-1307-4f8c-9c4e-9ef76586d99e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.023316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-50589c6e-1307-4f8c-9c4e-9ef76586d99e None None] MFA Rules not processed for user `15db5106efbd434f99a4d102810cc7a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:45.024537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-50589c6e-1307-4f8c-9c4e-9ef76586d99e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.165861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 700/1405] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:45.167187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.167473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.167747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.192221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005456c0-4b3b-470a-bf24-f2f5c9e20368 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:45.196664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:45.197817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.198302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.198666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:45.198987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.201546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.203041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.216651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:45.218671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:45.232609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 706/1406] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:45.234340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:45.236334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.305140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 701/1407] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:45.305140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:45.305140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.351011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1e181-2964-4802-960b-9f202acc1ccb service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:45.355071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:45.357517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.357517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.357517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.360325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.373462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:45.374878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:45.403163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 707/1408] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:45 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:45.404660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:45.409618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.451007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c435dd-73c2-4e4a-86f7-aa8218c51d8f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:45.464139 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1354de1a-a419-4212-bdd5-87340b310db5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:45.464785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354de1a-a419-4212-bdd5-87340b310db5 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.465076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354de1a-a419-4212-bdd5-87340b310db5 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.465378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354de1a-a419-4212-bdd5-87340b310db5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.467656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1354de1a-a419-4212-bdd5-87340b310db5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.545979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc644f4-53a4-408e-b8ec-d98a993a98ac service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:45.563715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:45.564647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.565250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.565715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.578478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.616051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:45.617746 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1354de1a-a419-4212-bdd5-87340b310db5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:45.619630 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1354de1a-a419-4212-bdd5-87340b310db5 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:45.620039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:45.634966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 702/1409] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:45.636296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31d3082-ceb8-45fd-925e-7466b5f5405c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.636587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31d3082-ceb8-45fd-925e-7466b5f5405c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.636865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31d3082-ceb8-45fd-925e-7466b5f5405c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.656865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 708/1410] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:45.659980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b31d3082-ceb8-45fd-925e-7466b5f5405c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:45.661252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b31d3082-ceb8-45fd-925e-7466b5f5405c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:45.671547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 703/1411] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:45.675518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:45.695351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f9f9f3-50e1-4f7c-b4ca-e66f75359e0a service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:45.700642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:45.726215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.726739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.727276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.728115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:45.728739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:45.730761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:45.763731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 709/1412] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:05:45.768429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31d3082-ceb8-45fd-925e-7466b5f5405c None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:45.782379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31d3082-ceb8-45fd-925e-7466b5f5405c None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:45.786935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:45.802397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:45.803185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:45.803473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:45.803749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:45.804392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:45.808366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:45.998300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 704/1413] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:45 2022] POST /identity/v3/projects => generated 373 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:45.999010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:46.013256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb3a171-7872-4114-a693-dd152f89ce3d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:46.107406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:46.108048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:46.108329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:46.108600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:46.109239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:46.111429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:46.207989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:46.259195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:46.273067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 710/1414] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:45 2022] POST /identity/v3/users => generated 287 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:46.277401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8134a98c-ce51-406c-8294-247d83f554c6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:46.283932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:46.284585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:46.284865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:46.285193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:46.285868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:46.287788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:46.310764 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:46.312091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 705/1415] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:46 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:46.336894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6019f6-79bc-4bf3-affc-eb17fccd4dba admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:46.342429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcebc4c2-268a-4079-a379-f26ce532d6d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:46.343431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcebc4c2-268a-4079-a379-f26ce532d6d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:46.343724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcebc4c2-268a-4079-a379-f26ce532d6d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:46.344009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcebc4c2-268a-4079-a379-f26ce532d6d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:46.344788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcebc4c2-268a-4079-a379-f26ce532d6d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:46.346452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcebc4c2-268a-4079-a379-f26ce532d6d9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:46.383225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:46.395949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 711/1416] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:46 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:46.406150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f72826a-708b-43fe-97d9-e48ed730b0cd admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:46.422490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec07a55e-d923-490a-bd63-b238b11b67b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:46.423262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec07a55e-d923-490a-bd63-b238b11b67b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:46.423533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec07a55e-d923-490a-bd63-b238b11b67b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:46.424065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec07a55e-d923-490a-bd63-b238b11b67b8 admin admin] PATH_INFO: `/v3/projects/9a81ed46a9bd45d2ba6958a878d6c1c3/users/78b885726648413e9b0dfd3f0ac8b858/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:46.448553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec07a55e-d923-490a-bd63-b238b11b67b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a81ed46a9bd45d2ba6958a878d6c1c3, user_id=78b885726648413e9b0dfd3f0ac8b858, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:46.457234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec07a55e-d923-490a-bd63-b238b11b67b8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:46.544696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 706/1417] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:46 2022] PUT /identity/v3/projects/9a81ed46a9bd45d2ba6958a878d6c1c3/users/78b885726648413e9b0dfd3f0ac8b858/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:46.556917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:46.557203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:46.557509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:46.630197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] MFA Rules not processed for user `78b885726648413e9b0dfd3f0ac8b858`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:46.631438 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:46.726178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:46.726671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:46.727085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:46.823892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] MFA Rules not processed for user `ce7f52b7f96c400fb7d5fb0453c94362`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:46.835658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:46.901449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 712/1418] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:46.904956 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:46.920941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:47.104754 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 707/1419] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:46 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 380 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:47.110767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:47.113200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ef9d9b-7715-4314-9450-70528d9844d7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:47.115901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:47.149540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:47.149540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:47.149540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:47.149540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] PATH_INFO: `/v3/users/a938c8acd4134f2dab23bc8c5dfc8d1f` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:47.158192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] RBAC: Authorizing `identity:delete_user(user_id=a938c8acd4134f2dab23bc8c5dfc8d1f)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:47.160198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:47.242891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94368866-91c4-4a2d-9b91-e62ed9b6f7b2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:47.259048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:47.260417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:47.261379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:47.262307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:47.299430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:47.308695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:47.335549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 708/1420] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:47 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:47.338253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:47.343919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:47.383115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3af3fc-ebe0-4dc3-938e-7ca83dce6f3c service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:47.388778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:47.389975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:47.390384 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:47.390733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] PATH_INFO: `/v3/users/2a21b8fb2dc54513bd5256eb05b19055` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:47.392533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a21b8fb2dc54513bd5256eb05b19055)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:47.394871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:47.509101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a938c8acd4134f2dab23bc8c5dfc8d1f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.584264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a938c8acd4134f2dab23bc8c5dfc8d1f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.628514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a21b8fb2dc54513bd5256eb05b19055'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.629599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a21b8fb2dc54513bd5256eb05b19055'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.697278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a21b8fb2dc54513bd5256eb05b19055'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.709779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a938c8acd4134f2dab23bc8c5dfc8d1f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.710430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a938c8acd4134f2dab23bc8c5dfc8d1f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.711427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a938c8acd4134f2dab23bc8c5dfc8d1f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.711891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a21b8fb2dc54513bd5256eb05b19055'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.712894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9902481-1db7-4698-9d10-c57df31677e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a21b8fb2dc54513bd5256eb05b19055'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:47.734186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 713/1421] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:46 2022] DELETE /identity/v3/users/a938c8acd4134f2dab23bc8c5dfc8d1f => generated 0 bytes in 832 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:47.735212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:47.738691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:47.838600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f722708-0794-45e4-a049-0dc2d2c559ff admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:47.844205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:47.844880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:47.845171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:47.845490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] PATH_INFO: `/v3/users/78b885726648413e9b0dfd3f0ac8b858` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:47.846659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78b885726648413e9b0dfd3f0ac8b858)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:47.848626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:47.860117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 709/1422] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:47 2022] DELETE /identity/v3/users/2a21b8fb2dc54513bd5256eb05b19055 => generated 0 bytes in 524 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:47.863130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948fadc-2515-44c4-bbd9-c586c0a40c6c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:47.863130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948fadc-2515-44c4-bbd9-c586c0a40c6c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:47.863130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948fadc-2515-44c4-bbd9-c586c0a40c6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:47.891376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5948fadc-2515-44c4-bbd9-c586c0a40c6c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:47.897518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5948fadc-2515-44c4-bbd9-c586c0a40c6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:48.035019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 710/1423] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:48.041586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:48.042296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:48.042866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:48.096225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:48.099239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:48.116050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 711/1424] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:48.126190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:48.132691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:48.134223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:48.210978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78b885726648413e9b0dfd3f0ac8b858'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:48.234435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78b885726648413e9b0dfd3f0ac8b858'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:48.323266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d37faa-8551-4de9-8e8e-bf7b7646b9b6 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:48.335308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b18c1ba-a20a-4b50-8cce-955d36fd9f99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:48.336204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b18c1ba-a20a-4b50-8cce-955d36fd9f99 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:48.336896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b18c1ba-a20a-4b50-8cce-955d36fd9f99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:48.337600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b18c1ba-a20a-4b50-8cce-955d36fd9f99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:48.374924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b18c1ba-a20a-4b50-8cce-955d36fd9f99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:48.380122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78b885726648413e9b0dfd3f0ac8b858'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:48.380771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78b885726648413e9b0dfd3f0ac8b858'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:48.381475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78b885726648413e9b0dfd3f0ac8b858'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:48.392456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b18c1ba-a20a-4b50-8cce-955d36fd9f99 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:48.416562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 714/1425] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:47 2022] DELETE /identity/v3/users/78b885726648413e9b0dfd3f0ac8b858 => generated 0 bytes in 681 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:48.422064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:48.429563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:48.450174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 712/1426] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:48.457120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe926c-beef-4018-a75f-52b6b58c60a1 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:48.458084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe926c-beef-4018-a75f-52b6b58c60a1 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:48.458634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe926c-beef-4018-a75f-52b6b58c60a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:48.475211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:48.506003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6bbe926c-beef-4018-a75f-52b6b58c60a1 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:48.509476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6bbe926c-beef-4018-a75f-52b6b58c60a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:48.571440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d69bf6-0f97-45b0-b951-478b1bcd0e05 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:48.576828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:48.577515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:48.577850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:48.578189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:48.580021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:48.632841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 713/1427] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:48.647554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbe926c-beef-4018-a75f-52b6b58c60a1 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:48.678906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbe926c-beef-4018-a75f-52b6b58c60a1 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:48.686704 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ef7940-1c7c-4433-a00e-a3ac285c50af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:48.687548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef7940-1c7c-4433-a00e-a3ac285c50af service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:48.687998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef7940-1c7c-4433-a00e-a3ac285c50af service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:48.688447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef7940-1c7c-4433-a00e-a3ac285c50af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:48.760402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef7940-1c7c-4433-a00e-a3ac285c50af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:48.761974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef7940-1c7c-4433-a00e-a3ac285c50af service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:48.782731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 714/1428] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:48.798483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:48.800074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:48.824825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:48.843088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 715/1429] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:49.168176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d8ce3-0aee-4d1c-8cd9-32b192f23bac None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:49.168738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d8ce3-0aee-4d1c-8cd9-32b192f23bac None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:49.168939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d8ce3-0aee-4d1c-8cd9-32b192f23bac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:49.235801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-bd4d8ce3-0aee-4d1c-8cd9-32b192f23bac None None] MFA Rules not processed for user `e0ba6bf2459d4a98a3a1f5684bdad2b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:49.246677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bd4d8ce3-0aee-4d1c-8cd9-32b192f23bac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:49.280755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:49.282367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:49.311766 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e6fd8-6e79-4416-afd1-957c1874b9a7 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:49.328959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:49.331462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:49.331927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:49.332942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] PATH_INFO: `/v3/projects/cd9d8e3aaecc4f0e9659b1c730f1aa7a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:49.339634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd9d8e3aaecc4f0e9659b1c730f1aa7a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:49.342876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:49.481799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 715/1430] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:49 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:49.487453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=87,keystone:INSERT=4,keystone:DELETE=9 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:49.613589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd9d8e3aaecc4f0e9659b1c730f1aa7a', 'request_id': 'req-f501e5da-a85f-4250-b3d6-37e4e287500c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:49.767365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd9d8e3aaecc4f0e9659b1c730f1aa7a', 'request_id': 'req-f501e5da-a85f-4250-b3d6-37e4e287500c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:49.777066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd9d8e3aaecc4f0e9659b1c730f1aa7a', 'request_id': 'req-f501e5da-a85f-4250-b3d6-37e4e287500c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:49.812880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] The token cache is being invalidate because project cd9d8e3aaecc4f0e9659b1c730f1aa7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:49.816606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:49.821940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:49.828794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 716/1431] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:49 2022] DELETE /identity/v3/projects/cd9d8e3aaecc4f0e9659b1c730f1aa7a => generated 0 bytes in 545 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:05:49.830058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4d8ce3-0aee-4d1c-8cd9-32b192f23bac None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:49.864015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4d8ce3-0aee-4d1c-8cd9-32b192f23bac None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:50.039780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:50.040057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:50.040642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:50.041125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] PATH_INFO: `/v3/users/1d213c0c12314059a31d6185d3136cab` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:50.042868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d213c0c12314059a31d6185d3136cab)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:50.045109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:50.110631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:50.112494 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:50.149340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f501e5da-a85f-4250-b3d6-37e4e287500c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:50.158389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:50.159331 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:50.159748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:50.160155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] PATH_INFO: `/v3/projects/aa35fbee77ab41d1a2db41f5f8bb39bf` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:50.162069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa35fbee77ab41d1a2db41f5f8bb39bf)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:50.164118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:50.263169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d213c0c12314059a31d6185d3136cab'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.264729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d213c0c12314059a31d6185d3136cab'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.294985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d213c0c12314059a31d6185d3136cab'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.313091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d213c0c12314059a31d6185d3136cab'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.314182 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d213c0c12314059a31d6185d3136cab'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.340049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa35fbee77ab41d1a2db41f5f8bb39bf', 'request_id': 'req-b00f8991-393f-4c7e-896e-c3aed6399d55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.391298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 716/1432] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:49 2022] DELETE /identity/v3/users/1d213c0c12314059a31d6185d3136cab => generated 0 bytes in 562 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:05:50.476033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa35fbee77ab41d1a2db41f5f8bb39bf', 'request_id': 'req-b00f8991-393f-4c7e-896e-c3aed6399d55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.486116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa35fbee77ab41d1a2db41f5f8bb39bf', 'request_id': 'req-b00f8991-393f-4c7e-896e-c3aed6399d55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.505022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:50.506540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:50.508462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] The token cache is being invalidate because project aa35fbee77ab41d1a2db41f5f8bb39bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:50.511577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.517926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:50.519745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 717/1433] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:50 2022] DELETE /identity/v3/projects/aa35fbee77ab41d1a2db41f5f8bb39bf => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:50.523319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:50.524943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:50.639051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:50.714377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f8991-393f-4c7e-896e-c3aed6399d55 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:50.725906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:50.735756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:50.735756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:50.736876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] PATH_INFO: `/v3/users/26b4805f8d934619ac02ab08b83ab6a4` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:50.740704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26b4805f8d934619ac02ab08b83ab6a4)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:50.749122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:50.752907 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b6b1f-e8f5-4c0a-9bfa-98b87af18e3c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:50.760374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:50.761523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:50.762005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:50.762595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] PATH_INFO: `/v3/projects/9a81ed46a9bd45d2ba6958a878d6c1c3` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:50.766593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a81ed46a9bd45d2ba6958a878d6c1c3)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:50.777801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:50.985493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a81ed46a9bd45d2ba6958a878d6c1c3', 'request_id': 'req-ad32769a-0df6-45e5-a123-e4484ed0f9bb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.002365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a81ed46a9bd45d2ba6958a878d6c1c3', 'request_id': 'req-ad32769a-0df6-45e5-a123-e4484ed0f9bb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.035474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a81ed46a9bd45d2ba6958a878d6c1c3', 'request_id': 'req-ad32769a-0df6-45e5-a123-e4484ed0f9bb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.089635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26b4805f8d934619ac02ab08b83ab6a4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.112405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26b4805f8d934619ac02ab08b83ab6a4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.112528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] The token cache is being invalidate because project 9a81ed46a9bd45d2ba6958a878d6c1c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:51.112528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.113351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ad32769a-0df6-45e5-a123-e4484ed0f9bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.117192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 717/1434] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:50 2022] DELETE /identity/v3/projects/9a81ed46a9bd45d2ba6958a878d6c1c3 => generated 0 bytes in 613 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:05:51.121845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:51.123053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:51.124073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:51.196731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:51.204552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:51.239033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26b4805f8d934619ac02ab08b83ab6a4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.239669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26b4805f8d934619ac02ab08b83ab6a4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.240590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea1d0927-995e-4891-8c51-9a67b5c2a073 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26b4805f8d934619ac02ab08b83ab6a4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:51.279295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 718/1435] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:05:50 2022] DELETE /identity/v3/users/26b4805f8d934619ac02ab08b83ab6a4 => generated 0 bytes in 756 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:05:51.410488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a7245e-e003-4085-a600-0f228cb6fac2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:51.410779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a7245e-e003-4085-a600-0f228cb6fac2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:51.411301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a7245e-e003-4085-a600-0f228cb6fac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:51.412055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 718/1436] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:51.487321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a2a7245e-e003-4085-a600-0f228cb6fac2 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:51.488759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a2a7245e-e003-4085-a600-0f228cb6fac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:51.515915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:51.517942 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:51.569614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e9b44-0c3f-474f-81ce-9c3cb29f975b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:51.627867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:51.710275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 719/1437] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:51.713941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cd7b43-2db6-4ca2-85b7-c410affe22fe None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:51.715037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cd7b43-2db6-4ca2-85b7-c410affe22fe None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:51.715648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cd7b43-2db6-4ca2-85b7-c410affe22fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:51.753027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:51.753267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:51.753578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:51.758720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] PATH_INFO: `/v3/projects/1e6622c4b6fe4dbda72b6b9702b32e3f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:51.766348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e6622c4b6fe4dbda72b6b9702b32e3f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:51.772947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:51.918468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-81cd7b43-2db6-4ca2-85b7-c410affe22fe None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:51.920849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-81cd7b43-2db6-4ca2-85b7-c410affe22fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:52.073710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e6622c4b6fe4dbda72b6b9702b32e3f', 'request_id': 'req-92f2314f-7b36-46cd-b450-4949248d78ff'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:52.110351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e6622c4b6fe4dbda72b6b9702b32e3f', 'request_id': 'req-92f2314f-7b36-46cd-b450-4949248d78ff'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:52.114944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 720/1438] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:05:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 404 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:52.115833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cd7b43-2db6-4ca2-85b7-c410affe22fe None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:52.136686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cd7b43-2db6-4ca2-85b7-c410affe22fe None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:52.177301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e6622c4b6fe4dbda72b6b9702b32e3f', 'request_id': 'req-92f2314f-7b36-46cd-b450-4949248d78ff'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:52.295993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:52.297243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:52.298421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:52.303234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:52.303234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:52.303234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:52.308261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] The token cache is being invalidate because project 1e6622c4b6fe4dbda72b6b9702b32e3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:52.308546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:52.308674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:52.311280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92f2314f-7b36-46cd-b450-4949248d78ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:52.314275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 719/1439] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:51 2022] DELETE /identity/v3/projects/1e6622c4b6fe4dbda72b6b9702b32e3f => generated 0 bytes in 799 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:05:52.329726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bda374-3868-4be1-9fce-0a5209c6008c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:52.330022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bda374-3868-4be1-9fce-0a5209c6008c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:52.330308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bda374-3868-4be1-9fce-0a5209c6008c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:52.339281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 721/1440] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:52.343402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:52.354351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:52.365853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-59bda374-3868-4be1-9fce-0a5209c6008c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:52.367180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-59bda374-3868-4be1-9fce-0a5209c6008c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:52.377360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 720/1441] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:52.386429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2c3b9-58cd-41c1-9640-4dc1dd9052fc None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:52.386868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2c3b9-58cd-41c1-9640-4dc1dd9052fc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:52.387300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2c3b9-58cd-41c1-9640-4dc1dd9052fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:52.393240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb3772-e9dd-416c-9fd0-fb05f5c0b0d0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:52.404701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:52.405495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:52.405786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:52.406072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:52.406717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:52.415613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:52.431662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-27d2c3b9-58cd-41c1-9640-4dc1dd9052fc None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:52.432892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-27d2c3b9-58cd-41c1-9640-4dc1dd9052fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:52.457916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 721/1442] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:52.459474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2c3b9-58cd-41c1-9640-4dc1dd9052fc None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:52.482443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2c3b9-58cd-41c1-9640-4dc1dd9052fc None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:52.497709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:52.499678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:52.499678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:52.499678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:52.501011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:52.502023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:52.508381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:52.555822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 722/1443] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:52 2022] POST /identity/v3/projects => generated 377 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:05:52.557036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:52.602828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 722/1444] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:52.607714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:52.643032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c27dfab-5aaf-469c-ae91-8921e2fc0a7e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:52.688251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:52.807654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b5b934-87be-440f-bb2c-51b3724b4c79 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:52.815350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:52.816574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:52.816689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:52.817056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:52.817532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:52.818294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:52.829407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:52.829853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:52.832790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:52.834352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:52.836377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:52.838467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.005576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 723/1445] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:52 2022] POST /identity/v3/projects => generated 381 bytes in 398 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:53.012084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 723/1446] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:52 2022] POST /identity/v3/users => generated 289 bytes in 455 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:53.013768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.015017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.023444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:53.027518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50aa5cf-6f11-45d3-af1f-c21f0da3f894 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.149528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.150166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.151670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.151958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.152621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.171581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.275844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc76baf5-181b-4cf9-b533-b7e97855ab99 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.279581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 724/1447] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:53 2022] POST /identity/v3/users => generated 291 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:53.281012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.283569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.284371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.284800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.286498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.291388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:53.325032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c042e2-67e9-4e56-a17f-e693ad4f0eb9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.339166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.339959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.340342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.340735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.342185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.344548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.379769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 725/1448] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:53 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:53.379769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.383320 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.385138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.406026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5f356a-57f9-46fe-bc4d-5c1233c18791 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.414862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 724/1449] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:05:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:05:53.416031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.432800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.434093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.436381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.436887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.437804 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.439709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.444801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83cba81-c454-4263-ae62-6a7a9b08d6a1 service swift] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.457853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.458695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.458981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.459255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.463029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.463029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.464862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 726/1450] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:53 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:53.472169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.487333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 725/1451] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:53 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:53.494736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.495976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d097e097-0b86-4920-a39f-1e90d8685808 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.510160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.514813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.514813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.514813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.514813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.517652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.521276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da1b58-ec44-4f52-ba89-2558d2b47b54 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.541658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 727/1452] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:53 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:53.545283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:53.558999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b30fbf3-cb05-49d8-afad-7251ee84d400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.559414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30fbf3-cb05-49d8-afad-7251ee84d400 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.559862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30fbf3-cb05-49d8-afad-7251ee84d400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.560851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30fbf3-cb05-49d8-afad-7251ee84d400 admin admin] PATH_INFO: `/v3/projects/643e3664a27c4eafb7fea2c0ee9d8f81/users/5969d9b178774b4ca799723b7bc78867/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.562384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6836a31-2d1c-40dc-8366-a9f215e692b7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:53.597486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b30fbf3-cb05-49d8-afad-7251ee84d400 admin admin] RBAC: Authorizing `identity:create_grant(project_id=643e3664a27c4eafb7fea2c0ee9d8f81, user_id=5969d9b178774b4ca799723b7bc78867, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.603076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b30fbf3-cb05-49d8-afad-7251ee84d400 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.607389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1c7d5-ec8a-4e67-a4b4-6c6cdacf21e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:53.608092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1c7d5-ec8a-4e67-a4b4-6c6cdacf21e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.608381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1c7d5-ec8a-4e67-a4b4-6c6cdacf21e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.608662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1c7d5-ec8a-4e67-a4b4-6c6cdacf21e6 admin admin] PATH_INFO: `/v3/projects/118a27afa541466c9125ed12bcc34977/users/52a768035a9e49b39768eb8b8f0787df/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.681090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1c7d5-ec8a-4e67-a4b4-6c6cdacf21e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=118a27afa541466c9125ed12bcc34977, user_id=52a768035a9e49b39768eb8b8f0787df, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:53.698811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1c7d5-ec8a-4e67-a4b4-6c6cdacf21e6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:53.707074 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 726/1453] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:53 2022] PUT /identity/v3/projects/643e3664a27c4eafb7fea2c0ee9d8f81/users/5969d9b178774b4ca799723b7bc78867/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:05:53.718801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.719261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.719724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.758547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:53.800916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] MFA Rules not processed for user `5969d9b178774b4ca799723b7bc78867`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:53.802223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:53.829825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 728/1454] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:53 2022] PUT /identity/v3/projects/118a27afa541466c9125ed12bcc34977/users/52a768035a9e49b39768eb8b8f0787df/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:53.842526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:53.842817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:53.843202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:53.853131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:53.948335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] MFA Rules not processed for user `52a768035a9e49b39768eb8b8f0787df`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:53.951811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:54.072349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 727/1455] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:54.075918 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:54.079618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:54.290924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 729/1456] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:53 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 449 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:54.292040 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:54.298424 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:54.409410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b78f51-a2a0-42fa-abbf-6654a6c526e4 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:54.554838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd9074b-b066-4720-ac5b-1c519ffd9201 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:54.561874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:54.562686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:54.562974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:54.563256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] PATH_INFO: `/v3/projects/69ee9bc5872241aa90e08f04008add89` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:54.565444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69ee9bc5872241aa90e08f04008add89)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:54.567848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:54.686392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:54.690171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:54.690593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:54.691260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:54.806120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69ee9bc5872241aa90e08f04008add89', 'request_id': 'req-5132afdf-9414-4937-9cd5-fe31c0405715'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:55.043727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:55.043727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:55.043727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 728/1457] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 918 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:55.043727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:55.059200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea6e88a-acf4-44ac-8edd-0e5360d76552 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:55.244079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69ee9bc5872241aa90e08f04008add89', 'request_id': 'req-5132afdf-9414-4937-9cd5-fe31c0405715'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:55.258836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69ee9bc5872241aa90e08f04008add89', 'request_id': 'req-5132afdf-9414-4937-9cd5-fe31c0405715'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:55.296582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:55.297185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:55.297495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:55.297769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:55.298414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:55.302432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:55.317400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] The token cache is being invalidate because project 69ee9bc5872241aa90e08f04008add89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:05:55.317920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:55.325747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5132afdf-9414-4937-9cd5-fe31c0405715 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:05:55.347070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 730/1458] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:05:54 2022] DELETE /identity/v3/projects/69ee9bc5872241aa90e08f04008add89 => generated 0 bytes in 1056 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:55.358924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e39094c-7b5f-4453-8ea2-27f54837ba6e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:55.359223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e39094c-7b5f-4453-8ea2-27f54837ba6e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:55.359503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e39094c-7b5f-4453-8ea2-27f54837ba6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:55.424001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3e39094c-7b5f-4453-8ea2-27f54837ba6e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:55.425396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3e39094c-7b5f-4453-8ea2-27f54837ba6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:55.454862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 731/1459] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:55.462075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a7888-16b6-4947-80cc-4a644c153810 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:55.463058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a7888-16b6-4947-80cc-4a644c153810 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:55.463689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a7888-16b6-4947-80cc-4a644c153810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:55.466843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 729/1460] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:54 2022] POST /identity/v3/projects => generated 383 bytes in 475 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:55.474168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:55.481533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:55.532018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-aa5a7888-16b6-4947-80cc-4a644c153810 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:55.542534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aa5a7888-16b6-4947-80cc-4a644c153810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:55.681102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4337199-56c8-4cb9-b5ca-8d7280e0439e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:55.718371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:55.725510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:55.725510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:55.725510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:55.725510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:55.725510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:55.806841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 732/1461] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:55.806841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4fbd94-6a0d-4de4-936f-00fa69457565 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:55.806841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4fbd94-6a0d-4de4-936f-00fa69457565 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:55.806841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4fbd94-6a0d-4de4-936f-00fa69457565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:55.870526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 730/1462] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:55 2022] POST /identity/v3/users => generated 298 bytes in 390 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:55.870526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:55.875102 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-dc4fbd94-6a0d-4de4-936f-00fa69457565 None None] MFA Rules not processed for user `91639dc527ae47e892164544572cbec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:55.877049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dc4fbd94-6a0d-4de4-936f-00fa69457565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:55.894769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158dc8d5-f382-4635-bc71-88d44ccf4207 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:55.912223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:55.913042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:55.913585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:55.914087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:55.916752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:05:55.917375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:55.922826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:55.968271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 731/1463] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:05:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:05:55.969598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:55.988417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8360109f-2805-4b5d-a7f4-7f72e1b9db81 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:56.010264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:56.014824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:56.029816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:56.051326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:56.052364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:56.059234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:56.067277 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:56.083812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:56.153067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 732/1464] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:55 2022] GET /identity/v3/roles => generated 1420 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:56.154540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:56.159883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 733/1465] 10.222.0.62 () {58 vars in 1015 bytes} [Fri Oct 28 20:05:55 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 361 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:56.166003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4fbd94-6a0d-4de4-936f-00fa69457565 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:56.226908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9860665-b59b-4fb3-845f-63b56bbbdf0a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:56.229466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4fbd94-6a0d-4de4-936f-00fa69457565 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:56.248985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:56.257202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:56.258188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:56.258352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:56.258850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:56.259960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:56.263565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:56.269361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] PATH_INFO: `/v3/projects/c745b90f844e4dfb85c00ff8c99fbb04/users/8f28df08258b45109c6a6f9e3d9eb9b2/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:56.273004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:56.277926 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:56.347458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c745b90f844e4dfb85c00ff8c99fbb04, user_id=8f28df08258b45109c6a6f9e3d9eb9b2, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:56.347458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:56.440172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 734/1466] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:56 2022] PUT /identity/v3/projects/c745b90f844e4dfb85c00ff8c99fbb04/users/8f28df08258b45109c6a6f9e3d9eb9b2/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:56.440978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 733/1467] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:56 2022] POST /identity/v3/projects => generated 357 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:56.441774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:56.446417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:56.449212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:56.459902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:56.639691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a07021-ce11-42d7-ab75-c28d46c608d9 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:56.648091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:56.648091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:56.648091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:56.648091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:56.682045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:56.685404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:56.711106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 734/1468] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:56.712259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:56.863829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd64e9b-a4d6-424a-bfd2-1ab328506a99 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:56.874112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:56.875026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:56.877171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:56.877825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:56.878835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:56.883179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:56.914588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 735/1469] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:56 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:56.915916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:56.942721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83824534-4208-43ac-8649-5e0b84086a69 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:56.961573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fccc2c9-1ceb-4579-8e8d-ff0b7443806b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:56.962885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:56.963523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:56.963811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:56.964088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:56.964731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:56.968204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:56.973565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:56.978752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:56.978752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:56.978752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:56.992498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:56.994429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:57.024771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 735/1470] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 584 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:57.031733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:57.054299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cec44d5-002e-4cc9-bfa1-d92fdefbc5c3 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:57.072691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7968ca-a1c9-48dc-8b0d-17bc8f009f98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:57.073887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7968ca-a1c9-48dc-8b0d-17bc8f009f98 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:57.074629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7968ca-a1c9-48dc-8b0d-17bc8f009f98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:57.077900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7968ca-a1c9-48dc-8b0d-17bc8f009f98 admin admin] PATH_INFO: `/v3/projects/c745b90f844e4dfb85c00ff8c99fbb04/users/8f28df08258b45109c6a6f9e3d9eb9b2/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:57.081445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7968ca-a1c9-48dc-8b0d-17bc8f009f98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c745b90f844e4dfb85c00ff8c99fbb04, user_id=8f28df08258b45109c6a6f9e3d9eb9b2, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:57.087973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 736/1471] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:56 2022] POST /identity/v3/users => generated 279 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:57.088491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7968ca-a1c9-48dc-8b0d-17bc8f009f98 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:57.093573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:57.117018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 736/1472] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:57 2022] PUT /identity/v3/projects/c745b90f844e4dfb85c00ff8c99fbb04/users/8f28df08258b45109c6a6f9e3d9eb9b2/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:57.131276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041c9be6-140d-43be-8bec-df8ca760315a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:57.133021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:57.133423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:57.133747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:57.220724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] MFA Rules not processed for user `8f28df08258b45109c6a6f9e3d9eb9b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:57.223324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:57.277658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:57.293073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:57.294359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:57.295032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:57.295431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:57.296363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:57.298343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:57.358459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 737/1473] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:57 2022] GET /identity/v3/roles => generated 1420 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:05:57.359552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:57.381924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a7b74-c7ff-4e6d-baf7-aab509f186f1 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:57.386215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:57.399699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:57.400492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:57.400914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:57.401289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:57.402072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:57.404901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:57.429861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 738/1474] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:57 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:57.434331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:57.459458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ece92-be3c-4ee1-8071-c52396821e4d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:57.465238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 737/1475] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:57 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:57.469778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:57.471395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d74d7f6-1c47-4ee3-932a-d485b7c7ace9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:57.472195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74d7f6-1c47-4ee3-932a-d485b7c7ace9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:57.472578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74d7f6-1c47-4ee3-932a-d485b7c7ace9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:57.472955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74d7f6-1c47-4ee3-932a-d485b7c7ace9 admin admin] PATH_INFO: `/v3/projects/72df80d266cf4e04915a5e7d2b556eaf/users/0763f168579b4e46b162240adcb48ba3/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:57.474278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:57.507253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d74d7f6-1c47-4ee3-932a-d485b7c7ace9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72df80d266cf4e04915a5e7d2b556eaf, user_id=0763f168579b4e46b162240adcb48ba3, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:57.510290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d74d7f6-1c47-4ee3-932a-d485b7c7ace9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:57.624687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 739/1476] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:57 2022] PUT /identity/v3/projects/72df80d266cf4e04915a5e7d2b556eaf/users/0763f168579b4e46b162240adcb48ba3/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:57.627183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90945ed-ddc3-4524-83bc-f3515f3e60c5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:57.627593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90945ed-ddc3-4524-83bc-f3515f3e60c5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:57.628030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90945ed-ddc3-4524-83bc-f3515f3e60c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:57.630026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7350326f-f93b-4e41-9a0c-0a6f9af3b4af None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:57.700906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a90945ed-ddc3-4524-83bc-f3515f3e60c5 None None] MFA Rules not processed for user `0763f168579b4e46b162240adcb48ba3`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:57.702729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a90945ed-ddc3-4524-83bc-f3515f3e60c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:57.816710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:57.818654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:57.818952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:57.824169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:57.920128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 740/1477] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:57 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:57.924010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90945ed-ddc3-4524-83bc-f3515f3e60c5 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:58.048624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:58.053825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:58.081095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 738/1478] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:57 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 613 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:58.137989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90945ed-ddc3-4524-83bc-f3515f3e60c5 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:58.144747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:58.145524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:58.145810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:58.146137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:58.147045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:58.151197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:58.201674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:58.248316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:58.270219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:58.293123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4208ddd5-f24a-4b19-81ba-7bbb7af7e22c service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:58.300189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 741/1479] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:05:57 2022] POST /identity/v3/projects => generated 357 bytes in 372 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:58.300189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:58.402312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e5575-c92d-4267-bce1-55727b2c0645 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:58.409022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:58.410023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:58.410459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:58.410987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:58.411803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:58.415715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:58.417375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:58.418086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:58.418442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:58.418651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:58.433073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:58.539629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:58.541273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:58.567625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 742/1480] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:05:58 2022] POST /identity/v3/users => generated 285 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:05:58.581809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:58.583813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 739/1481] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:58 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:05:58.630586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a4c013-15e5-4da4-bcbe-fbcc508e58f8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:58.637129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:58.638774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:58.639795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:58.639795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:58.640646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:58.642990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:58.665911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:58.668886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 743/1482] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:58.681983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7d6a69-f673-403f-8ef5-107033e65c36 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:58.689458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:58.690838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:58.693748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:58.694260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] PATH_INFO: `/v3/projects/4a655235a6b542228be1db9ddaf4ad7f/users/7f9d85665b7e4e2e94ac3e5f68f0dbbe/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:58.720595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a655235a6b542228be1db9ddaf4ad7f, user_id=7f9d85665b7e4e2e94ac3e5f68f0dbbe, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:58.726253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:58.774249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:58.802289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 740/1483] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:58 2022] PUT /identity/v3/projects/4a655235a6b542228be1db9ddaf4ad7f/users/7f9d85665b7e4e2e94ac3e5f68f0dbbe/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:58.807446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:58.826380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf3e0e8-e12e-450d-a7e7-6088ca077bd0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:59.034673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5c4fa1-b22d-463c-905a-e9254b2ef7b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:59.035286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5c4fa1-b22d-463c-905a-e9254b2ef7b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.035569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5c4fa1-b22d-463c-905a-e9254b2ef7b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.036010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5c4fa1-b22d-463c-905a-e9254b2ef7b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.036839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5c4fa1-b22d-463c-905a-e9254b2ef7b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:59.038721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5c4fa1-b22d-463c-905a-e9254b2ef7b2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:59.061440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 744/1484] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:05:59.069137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:59.087264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:05:59.089967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbab989-615b-4770-a5bf-764b2ad7e12e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:59.096235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:59.098412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.098412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.098412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] PATH_INFO: `/v3/projects/4a655235a6b542228be1db9ddaf4ad7f/users/7f9d85665b7e4e2e94ac3e5f68f0dbbe/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.110220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a655235a6b542228be1db9ddaf4ad7f, user_id=7f9d85665b7e4e2e94ac3e5f68f0dbbe, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:59.114509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:59.150045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 741/1485] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:05:59 2022] PUT /identity/v3/projects/4a655235a6b542228be1db9ddaf4ad7f/users/7f9d85665b7e4e2e94ac3e5f68f0dbbe/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:05:59.164293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2830167-1635-4d7f-94b6-5c18273ec771 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.164585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2830167-1635-4d7f-94b6-5c18273ec771 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.164865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2830167-1635-4d7f-94b6-5c18273ec771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.213863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e2830167-1635-4d7f-94b6-5c18273ec771 None None] MFA Rules not processed for user `7f9d85665b7e4e2e94ac3e5f68f0dbbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:59.215522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e2830167-1635-4d7f-94b6-5c18273ec771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:59.355357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 745/1486] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:05:59 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:05:59.380080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:59.385712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:59.461564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298566bc-db63-40bb-a6dc-2cb923426ce4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:59.466312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc072c57-2ce9-47ff-8efa-e013406d20da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:59.467056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc072c57-2ce9-47ff-8efa-e013406d20da service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.467355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc072c57-2ce9-47ff-8efa-e013406d20da service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.467623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc072c57-2ce9-47ff-8efa-e013406d20da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.477731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc072c57-2ce9-47ff-8efa-e013406d20da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:59.479272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc072c57-2ce9-47ff-8efa-e013406d20da service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:59.490288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 742/1487] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:59 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:59.522534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78185b84-2729-4721-974e-d88651ee14d0 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.524963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78185b84-2729-4721-974e-d88651ee14d0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.524963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78185b84-2729-4721-974e-d88651ee14d0 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.524963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 746/1488] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:05:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:05:59.564772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6f339-068d-4909-b310-76003d336165 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.565065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6f339-068d-4909-b310-76003d336165 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.565410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6f339-068d-4909-b310-76003d336165 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.600913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-aac6f339-068d-4909-b310-76003d336165 None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:05:59.602895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aac6f339-068d-4909-b310-76003d336165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:59.729060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 743/1489] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:05:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:05:59.745301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78185b84-2729-4721-974e-d88651ee14d0 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:59.746783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-78185b84-2729-4721-974e-d88651ee14d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:05:59.760371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6f339-068d-4909-b310-76003d336165 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:05:59.774757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6f339-068d-4909-b310-76003d336165 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:59.783795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78185b84-2729-4721-974e-d88651ee14d0 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:05:59.784026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0d6e27-65c8-43f4-85a2-11668f271fca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:59.784026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0d6e27-65c8-43f4-85a2-11668f271fca service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.784026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0d6e27-65c8-43f4-85a2-11668f271fca service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.784026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0d6e27-65c8-43f4-85a2-11668f271fca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.787968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e22416-3b31-466d-b418-f907291946ff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:05:59.788740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e22416-3b31-466d-b418-f907291946ff service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:05:59.789033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e22416-3b31-466d-b418-f907291946ff service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:05:59.789454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e22416-3b31-466d-b418-f907291946ff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:05:59.808115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e22416-3b31-466d-b418-f907291946ff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:59.810702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e22416-3b31-466d-b418-f907291946ff service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:59.842105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 747/1490] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:05:59.887487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0d6e27-65c8-43f4-85a2-11668f271fca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:05:59.889666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0d6e27-65c8-43f4-85a2-11668f271fca service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:05:59.921019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 744/1491] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:05:59 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:05:59.939877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:00.540808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114fdbd6-1cef-4de4-ad96-8879734f6d6a None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:00.541457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114fdbd6-1cef-4de4-ad96-8879734f6d6a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:00.541457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114fdbd6-1cef-4de4-ad96-8879734f6d6a None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:00.543124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 748/1492] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:06:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:06:00.568305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b6cdd1-5fdd-434b-b531-2fe9d11626ab None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:00.569195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b6cdd1-5fdd-434b-b531-2fe9d11626ab None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:00.569517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b6cdd1-5fdd-434b-b531-2fe9d11626ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:00.590937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f1b6cdd1-5fdd-434b-b531-2fe9d11626ab None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:00.592386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f1b6cdd1-5fdd-434b-b531-2fe9d11626ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:00.602730 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 745/1493] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:06:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:00.617025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114fdbd6-1cef-4de4-ad96-8879734f6d6a None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:00.632180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114fdbd6-1cef-4de4-ad96-8879734f6d6a None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:00.637892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e730ad-04e7-4740-bfdb-c7f6a6a642c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:00.639930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e730ad-04e7-4740-bfdb-c7f6a6a642c6 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:00.639930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e730ad-04e7-4740-bfdb-c7f6a6a642c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:00.639930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e730ad-04e7-4740-bfdb-c7f6a6a642c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:00.648463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e730ad-04e7-4740-bfdb-c7f6a6a642c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:00.650225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e730ad-04e7-4740-bfdb-c7f6a6a642c6 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:00.662397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 749/1494] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:00.734024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:00.818642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:01.412092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:06.859895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b6cdd1-5fdd-434b-b531-2fe9d11626ab None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:06.881383 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:06.995775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b6cdd1-5fdd-434b-b531-2fe9d11626ab None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:07.002915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:07.004820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:07.004820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:07.004820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] PATH_INFO: `/v3/users/52a768035a9e49b39768eb8b8f0787df` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:07.006636 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] RBAC: Authorizing `identity:delete_user(user_id=52a768035a9e49b39768eb8b8f0787df)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:07.009073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:07.218521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52a768035a9e49b39768eb8b8f0787df'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:07.233718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52a768035a9e49b39768eb8b8f0787df'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:07.295290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52a768035a9e49b39768eb8b8f0787df'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:07.296277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52a768035a9e49b39768eb8b8f0787df'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:07.298401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52a768035a9e49b39768eb8b8f0787df'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:07.323492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 746/1495] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:06 2022] DELETE /identity/v3/users/52a768035a9e49b39768eb8b8f0787df => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:07.328993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:07.329541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:07.329968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:07.353689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:07.355000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:07.435173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 750/1496] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:07.479880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:07.481749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:07.572847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a78334e-a763-4b1e-8351-299eaf07541a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:07.578942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:07.583150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:07.583150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:07.583150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:07.596199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:07.598008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:07.610052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 747/1497] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:09.255664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:09.257094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:09.298615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e580cdb9-084e-4443-b32c-bf78649be5e0 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:09.329273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:09.330217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:09.331300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:09.331734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] PATH_INFO: `/v3/projects/118a27afa541466c9125ed12bcc34977` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:09.333296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] RBAC: Authorizing `identity:delete_project(project_id=118a27afa541466c9125ed12bcc34977)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:09.337527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:09.397212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:09.537031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '118a27afa541466c9125ed12bcc34977', 'request_id': 'req-11cbb114-3284-4cfc-a575-88f86dd43570'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:09.544593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '118a27afa541466c9125ed12bcc34977', 'request_id': 'req-11cbb114-3284-4cfc-a575-88f86dd43570'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:09.553099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '118a27afa541466c9125ed12bcc34977', 'request_id': 'req-11cbb114-3284-4cfc-a575-88f86dd43570'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:09.687372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] The token cache is being invalidate because project 118a27afa541466c9125ed12bcc34977 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:09.687677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:09.688353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:09.689995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 751/1498] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:09 2022] DELETE /identity/v3/projects/118a27afa541466c9125ed12bcc34977 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:09.699278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:09.723580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ea89ee-e0e2-4706-8eea-21abbedfee23 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:09.768705 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:09.801155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:09.801933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:09.802223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:09.802675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] PATH_INFO: `/v3/users/8f28df08258b45109c6a6f9e3d9eb9b2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:09.803825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f28df08258b45109c6a6f9e3d9eb9b2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:09.805969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:10.003746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f28df08258b45109c6a6f9e3d9eb9b2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:10.020333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f28df08258b45109c6a6f9e3d9eb9b2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:10.105659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f28df08258b45109c6a6f9e3d9eb9b2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:10.107295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f28df08258b45109c6a6f9e3d9eb9b2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:10.107947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-debb79ea-7b0d-4d46-b16f-9eb5aee54146 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f28df08258b45109c6a6f9e3d9eb9b2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:10.120940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 748/1499] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:09 2022] DELETE /identity/v3/users/8f28df08258b45109c6a6f9e3d9eb9b2 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:11.043925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:11.046251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:11.062017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbb114-3284-4cfc-a575-88f86dd43570 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:11.131953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:11.132613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:11.132904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:11.133178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] PATH_INFO: `/v3/projects/c745b90f844e4dfb85c00ff8c99fbb04` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:11.134512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c745b90f844e4dfb85c00ff8c99fbb04)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:11.138564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:11.280172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c745b90f844e4dfb85c00ff8c99fbb04', 'request_id': 'req-91e9edd3-e2b9-4893-93b4-4a58949b287a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:11.288504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c745b90f844e4dfb85c00ff8c99fbb04', 'request_id': 'req-91e9edd3-e2b9-4893-93b4-4a58949b287a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:11.298629 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c745b90f844e4dfb85c00ff8c99fbb04', 'request_id': 'req-91e9edd3-e2b9-4893-93b4-4a58949b287a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:11.396524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] The token cache is being invalidate because project c745b90f844e4dfb85c00ff8c99fbb04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:11.396977 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:11.397908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-91e9edd3-e2b9-4893-93b4-4a58949b287a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:11.402467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 752/1500] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:11 2022] DELETE /identity/v3/projects/c745b90f844e4dfb85c00ff8c99fbb04 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:11.408250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cfb942-1c64-4e99-8716-71c6e0936e4f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:11.408788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cfb942-1c64-4e99-8716-71c6e0936e4f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:11.408870 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cfb942-1c64-4e99-8716-71c6e0936e4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:11.439915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-02cfb942-1c64-4e99-8716-71c6e0936e4f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:11.441560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-02cfb942-1c64-4e99-8716-71c6e0936e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:11.545993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 749/1501] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:11.548173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10d9731-040f-4940-8b65-28a51d89dde1 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:11.548500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10d9731-040f-4940-8b65-28a51d89dde1 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:11.549772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10d9731-040f-4940-8b65-28a51d89dde1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:11.614627 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e10d9731-040f-4940-8b65-28a51d89dde1 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:11.617588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e10d9731-040f-4940-8b65-28a51d89dde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:11.638227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 753/1502] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:11.640372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cfb942-1c64-4e99-8716-71c6e0936e4f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:11.660169 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cfb942-1c64-4e99-8716-71c6e0936e4f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:11.674417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:11.675129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:11.675445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:11.675725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:11.676776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:11.677237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:11.681958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:11.744453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 750/1503] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:11.748349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10d9731-040f-4940-8b65-28a51d89dde1 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:11.763107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10d9731-040f-4940-8b65-28a51d89dde1 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:11.773285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:11.775127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:11.776075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:11.776361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:11.777179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:11.780441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:12.753734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 754/1504] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:11 2022] POST /identity/v3/projects => generated 389 bytes in 1007 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:12.758387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:12.769427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e856155e-c064-443b-9a03-5f8435347017 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:12.820574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:12.860876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:12.861533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:12.861817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:12.862096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:12.862725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:12.864633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:12.949766 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 751/1505] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:12 2022] POST /identity/v3/users => generated 295 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:12.956946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:12.965208 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e1700-dc1b-4dbb-a396-0a2b1a258d0a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:12.970469 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:12.971277 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:12.971571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:12.971842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:12.972703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:12.974218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:12.980316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:12.985836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 755/1506] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:12.991056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.006474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24ede13-016c-4fc6-bf61-ba42173af9fa admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.011324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f1c890-7810-417c-b92f-514aed27c4f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.011930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1c890-7810-417c-b92f-514aed27c4f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.012211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1c890-7810-417c-b92f-514aed27c4f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.012501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1c890-7810-417c-b92f-514aed27c4f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.013371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f1c890-7810-417c-b92f-514aed27c4f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:13.014968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f1c890-7810-417c-b92f-514aed27c4f2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:13.031784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 752/1507] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:12 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:13.031964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.043345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a5cf9-3e0c-4490-926f-73b45d729fcb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.049003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.049773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.050120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.050402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] PATH_INFO: `/v3/projects/3c7a17e332544953a1d818a1b5076bd0/users/14457482f4c24f44aed89dcb369f3991/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.068063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c7a17e332544953a1d818a1b5076bd0, user_id=14457482f4c24f44aed89dcb369f3991, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:13.070752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:13.151718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 756/1508] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:13 2022] PUT /identity/v3/projects/3c7a17e332544953a1d818a1b5076bd0/users/14457482f4c24f44aed89dcb369f3991/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:13.162719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0d5e5-f68e-4ff9-8719-a5addee3dd9a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.163028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0d5e5-f68e-4ff9-8719-a5addee3dd9a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.163307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0d5e5-f68e-4ff9-8719-a5addee3dd9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.195846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-36e0d5e5-f68e-4ff9-8719-a5addee3dd9a None None] MFA Rules not processed for user `14457482f4c24f44aed89dcb369f3991`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:13.197113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-36e0d5e5-f68e-4ff9-8719-a5addee3dd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:13.309488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 753/1509] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:13 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:13.311395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.376502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381bed2b-1124-417b-a851-bf72c1ab4911 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.381533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.382262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.382447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.382720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.383344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:13.385245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:13.517863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 757/1510] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:13 2022] POST /identity/v3/projects => generated 385 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:13.521907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e0d5e5-f68e-4ff9-8719-a5addee3dd9a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.532865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e0d5e5-f68e-4ff9-8719-a5addee3dd9a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.591807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.592436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.592719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.592995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.593789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:13.595713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:13.681884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 754/1511] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:13 2022] POST /identity/v3/users => generated 299 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:13.686541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.696697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a46239-ba2d-40d3-bb64-d3555bbf7fbe admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.702298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.703427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.703427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.703427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.704059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:13.705687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:13.717654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 758/1512] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:13.721898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.732395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1a8f7b-a307-418d-a9ea-c354f7d6a4c6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.739553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.740312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.740717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.741129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] PATH_INFO: `/v3/projects/3ae9aeafbe414b2dacb82eacf6a79748/users/54c7099848bc4a59a0db35399a59a315/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.758777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ae9aeafbe414b2dacb82eacf6a79748, user_id=54c7099848bc4a59a0db35399a59a315, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:13.761748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:13.865628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 755/1513] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:13 2022] PUT /identity/v3/projects/3ae9aeafbe414b2dacb82eacf6a79748/users/54c7099848bc4a59a0db35399a59a315/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:13.869547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.881493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736bd212-b46b-4639-a590-1beef174fff4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.963894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df26e01-254e-4661-a463-4eaab5a21d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.964519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df26e01-254e-4661-a463-4eaab5a21d7e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.964805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df26e01-254e-4661-a463-4eaab5a21d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.965106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df26e01-254e-4661-a463-4eaab5a21d7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:13.966441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df26e01-254e-4661-a463-4eaab5a21d7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:13.968086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df26e01-254e-4661-a463-4eaab5a21d7e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:13.981782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 759/1514] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:13 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:13.985409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:13.993098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac11349-3ec5-4427-be52-a08da54a1a84 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:13.998448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:13.999099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:13.999381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:13.999714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] PATH_INFO: `/v3/projects/3ae9aeafbe414b2dacb82eacf6a79748/users/54c7099848bc4a59a0db35399a59a315/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:14.001609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ae9aeafbe414b2dacb82eacf6a79748, user_id=54c7099848bc4a59a0db35399a59a315, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:14.004217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:14.009599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:14.014773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 756/1515] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:13 2022] PUT /identity/v3/projects/3ae9aeafbe414b2dacb82eacf6a79748/users/54c7099848bc4a59a0db35399a59a315/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:14.030124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c59e11f-3d3a-4b26-8d70-485e265cf922 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:14.030860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c59e11f-3d3a-4b26-8d70-485e265cf922 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:14.031437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c59e11f-3d3a-4b26-8d70-485e265cf922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:14.062738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6c59e11f-3d3a-4b26-8d70-485e265cf922 None None] MFA Rules not processed for user `54c7099848bc4a59a0db35399a59a315`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:14.063934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6c59e11f-3d3a-4b26-8d70-485e265cf922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:14.139689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 760/1516] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:14 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:14.150475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:14.152281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:14.167120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:14.214056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09be89cf-bf71-4847-bab5-b312bcff3f90 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:14.218393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4c92c3-59ac-4d1b-9951-0c8059d2162a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:14.219013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4c92c3-59ac-4d1b-9951-0c8059d2162a service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:14.219298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4c92c3-59ac-4d1b-9951-0c8059d2162a service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:14.219577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4c92c3-59ac-4d1b-9951-0c8059d2162a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:14.228925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4c92c3-59ac-4d1b-9951-0c8059d2162a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:14.230378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4c92c3-59ac-4d1b-9951-0c8059d2162a service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:14.241341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 757/1517] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:14 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:14.382242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c59e11f-3d3a-4b26-8d70-485e265cf922 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:14.457425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c59e11f-3d3a-4b26-8d70-485e265cf922 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:14.462002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:14.462639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:14.462924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:14.463205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] PATH_INFO: `/v3/users/14457482f4c24f44aed89dcb369f3991` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:14.464328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14457482f4c24f44aed89dcb369f3991)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:14.467276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:14.620720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14457482f4c24f44aed89dcb369f3991'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:14.621405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14457482f4c24f44aed89dcb369f3991'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:14.633596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14457482f4c24f44aed89dcb369f3991'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:14.641054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14457482f4c24f44aed89dcb369f3991'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:14.641731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14457482f4c24f44aed89dcb369f3991'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:14.710354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 761/1518] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:14 2022] DELETE /identity/v3/users/14457482f4c24f44aed89dcb369f3991 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:14.715161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:14.715595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:14.716608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:14.736030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:14.737227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:14.800628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 758/1519] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:14.812482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:14.815541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:14.879201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92805ad8-0a41-4d94-8558-018e72ad3e20 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:14.883203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:14.883825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:14.884111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:14.884388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:14.890970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:14.895040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:14.896531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:14.906726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 762/1520] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:15.317823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:15.319353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:15.333461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5e80a3-a1c1-45d0-82f9-8264967a2067 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:15.338793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:15.339594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:15.339889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:15.340170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] PATH_INFO: `/v3/projects/3c7a17e332544953a1d818a1b5076bd0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:15.341425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c7a17e332544953a1d818a1b5076bd0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:15.343435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:15.443867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c7a17e332544953a1d818a1b5076bd0', 'request_id': 'req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.492658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c7a17e332544953a1d818a1b5076bd0', 'request_id': 'req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.500272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c7a17e332544953a1d818a1b5076bd0', 'request_id': 'req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.511051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] The token cache is being invalidate because project 3c7a17e332544953a1d818a1b5076bd0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:15.511341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.511983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.513670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 759/1521] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:15 2022] DELETE /identity/v3/projects/3c7a17e332544953a1d818a1b5076bd0 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:15.517671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:15.527252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0f2e50-9a75-4db9-a082-bc9f530eb1b6 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:15.584976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:15.585661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:15.585937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:15.586240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] PATH_INFO: `/v3/users/54c7099848bc4a59a0db35399a59a315` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:15.587369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=54c7099848bc4a59a0db35399a59a315)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:15.589304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:15.804087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54c7099848bc4a59a0db35399a59a315'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.806443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54c7099848bc4a59a0db35399a59a315'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.821886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54c7099848bc4a59a0db35399a59a315'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.859339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54c7099848bc4a59a0db35399a59a315'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:15.867222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f36ee115-b714-4f16-bfa2-86e2a799a6ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54c7099848bc4a59a0db35399a59a315'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:16.038222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 763/1522] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:15 2022] DELETE /identity/v3/users/54c7099848bc4a59a0db35399a59a315 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:16.484981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:16.492377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:16.514914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca68fc0e-81b3-4f32-96cb-57e4a65dbf21 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:16.543244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:16.594144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:16.594834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:16.595069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:16.595364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] PATH_INFO: `/v3/projects/3ae9aeafbe414b2dacb82eacf6a79748` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:16.596771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ae9aeafbe414b2dacb82eacf6a79748)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:16.598758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:16.727696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ae9aeafbe414b2dacb82eacf6a79748', 'request_id': 'req-4c997960-f5f6-416c-a400-8462c2d28e7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:16.784873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ae9aeafbe414b2dacb82eacf6a79748', 'request_id': 'req-4c997960-f5f6-416c-a400-8462c2d28e7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:16.791946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ae9aeafbe414b2dacb82eacf6a79748', 'request_id': 'req-4c997960-f5f6-416c-a400-8462c2d28e7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:16.804885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] The token cache is being invalidate because project 3ae9aeafbe414b2dacb82eacf6a79748 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:16.805184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:16.805907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c997960-f5f6-416c-a400-8462c2d28e7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:16.807411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 760/1523] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:16 2022] DELETE /identity/v3/projects/3ae9aeafbe414b2dacb82eacf6a79748 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:16.812527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e010d3-37ac-4c72-8c31-ecee41b51e0c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:16.812821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e010d3-37ac-4c72-8c31-ecee41b51e0c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:16.813099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e010d3-37ac-4c72-8c31-ecee41b51e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:16.834283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-54e010d3-37ac-4c72-8c31-ecee41b51e0c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:16.835488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-54e010d3-37ac-4c72-8c31-ecee41b51e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:16.899969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 764/1524] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:16.909948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afe9b5d-313a-46ff-9095-f6e318e99008 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:16.909948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afe9b5d-313a-46ff-9095-f6e318e99008 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:16.910189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afe9b5d-313a-46ff-9095-f6e318e99008 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:16.931117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9afe9b5d-313a-46ff-9095-f6e318e99008 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:16.932315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9afe9b5d-313a-46ff-9095-f6e318e99008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:16.943689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 761/1525] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:16.945355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e010d3-37ac-4c72-8c31-ecee41b51e0c None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:16.955628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e010d3-37ac-4c72-8c31-ecee41b51e0c None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:16.960523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:16.961134 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:16.961445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:16.961724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:16.962369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:16.962671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:16.964275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:16.975669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:16.979964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 765/1526] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:16.984895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afe9b5d-313a-46ff-9095-f6e318e99008 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:16.995388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afe9b5d-313a-46ff-9095-f6e318e99008 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:17.000628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:17.003961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.003961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.003961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.003961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:17.004545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:17.108589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 762/1527] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:16 2022] POST /identity/v3/projects => generated 385 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:17.113479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:17.128414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de646a2f-f5a2-4f90-ba7d-cf7c5f33a2f4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:17.208464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:17.209179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.209604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.209922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.210802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:17.213368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:17.299929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 766/1528] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:17 2022] POST /identity/v3/users => generated 293 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:17.304054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:17.315591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db951047-acff-455b-9dde-f5a90aa81af6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:17.321122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:17.322318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.322601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.322876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.323762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:17.326070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:17.337850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 763/1529] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:17 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:17.344459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:17.353700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc32ad43-4e5b-4cd8-8b1c-1c1d5ff27d73 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:17.359599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f276290a-29e5-4821-87a8-06d327b714b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:17.360237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276290a-29e5-4821-87a8-06d327b714b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.360527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276290a-29e5-4821-87a8-06d327b714b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.360805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276290a-29e5-4821-87a8-06d327b714b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.363772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f276290a-29e5-4821-87a8-06d327b714b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:17.363772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f276290a-29e5-4821-87a8-06d327b714b4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:17.377169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 767/1530] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:17.378793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:17.390633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fab519-cba0-4fb4-8a0c-01c5292bd786 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:17.395553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:17.396377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.397020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.397020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] PATH_INFO: `/v3/projects/4582d981020b465daedbc2501e7ebe67/users/8883e606252a4cc09084e57fa871597b/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.417435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4582d981020b465daedbc2501e7ebe67, user_id=8883e606252a4cc09084e57fa871597b, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:17.420744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:17.472673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 764/1531] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:17 2022] PUT /identity/v3/projects/4582d981020b465daedbc2501e7ebe67/users/8883e606252a4cc09084e57fa871597b/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:17.482977 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de25e8-89f4-466e-b1ab-434f42c76ab1 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.483271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de25e8-89f4-466e-b1ab-434f42c76ab1 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.483553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de25e8-89f4-466e-b1ab-434f42c76ab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.516818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a0de25e8-89f4-466e-b1ab-434f42c76ab1 None None] MFA Rules not processed for user `8883e606252a4cc09084e57fa871597b`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:17.518099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a0de25e8-89f4-466e-b1ab-434f42c76ab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:17.629723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 768/1532] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:17 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:17.635559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:17.717787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:17.734306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc57ba-be81-4bd9-a396-8ad250d3f49a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:17.740239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:17.743853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.743853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.743853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.743853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:17.744150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:17.802197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9430854d-4bbd-468c-a6ea-6ba45750ab63 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.802592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9430854d-4bbd-468c-a6ea-6ba45750ab63 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.802798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9430854d-4bbd-468c-a6ea-6ba45750ab63 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.804720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 769/1533] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:06:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:06:17.829444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.829874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.830210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.859840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:17.861519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:17.870492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 765/1534] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:17 2022] POST /identity/v3/projects => generated 385 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:17.874984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:17.947001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:17.959590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 770/1535] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:06:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:17.963300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cd6b39-40f2-4831-b659-462957296a8f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:17.969491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:17.970410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:17.971168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:17.971785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:17.972679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:17.974916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:17.975051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:17.976747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:17.992251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc7a706-295e-4964-9c0c-76ee68d64fb7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:18.096055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 766/1536] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:17 2022] POST /identity/v3/users => generated 299 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:18.098315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:18.115576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c39b2-f78d-4ed9-9165-9496c66cc089 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:18.116286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:18.117967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:18.118379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:18.118772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:18.121502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:18.122842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:18.123154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:18.123433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:18.124081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:18.126214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:18.129925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:18.131632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:18.141358 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 767/1537] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:18 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:18.144052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:18.145036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 771/1538] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:18.160820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb82d3-89c5-4cad-85b5-f982275f4053 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:18.166876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:18.167759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:18.168111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:18.168461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] PATH_INFO: `/v3/projects/5a61d3b9dce040d48d21149b9e48e159/users/aae54952adc646969464f2c74d2ef61b/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:18.193775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a61d3b9dce040d48d21149b9e48e159, user_id=aae54952adc646969464f2c74d2ef61b, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:18.199464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:18.288436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 768/1539] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:18 2022] PUT /identity/v3/projects/5a61d3b9dce040d48d21149b9e48e159/users/aae54952adc646969464f2c74d2ef61b/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:18.293711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:18.310549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105f15f4-0b80-42ac-806a-0a560ad0a657 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:18.392716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0445748e-4301-4379-a9e7-f4183ea868ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:18.393437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0445748e-4301-4379-a9e7-f4183ea868ec admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:18.393723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0445748e-4301-4379-a9e7-f4183ea868ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:18.394003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0445748e-4301-4379-a9e7-f4183ea868ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:18.394884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0445748e-4301-4379-a9e7-f4183ea868ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:18.396675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0445748e-4301-4379-a9e7-f4183ea868ec admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:18.421040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 772/1540] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:18 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:18.422327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:18.436861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91546256-85d7-4128-91da-1f84cd4f0e80 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:18.442719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:18.443411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:18.443695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:18.443983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] PATH_INFO: `/v3/projects/5a61d3b9dce040d48d21149b9e48e159/users/aae54952adc646969464f2c74d2ef61b/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:18.446103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a61d3b9dce040d48d21149b9e48e159, user_id=aae54952adc646969464f2c74d2ef61b, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:18.450706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:18.463032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 769/1541] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:18 2022] PUT /identity/v3/projects/5a61d3b9dce040d48d21149b9e48e159/users/aae54952adc646969464f2c74d2ef61b/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:18.475169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c92d953-7ae8-4b9e-a113-373d88615af2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:18.475551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c92d953-7ae8-4b9e-a113-373d88615af2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:18.475905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c92d953-7ae8-4b9e-a113-373d88615af2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:18.521112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-0c92d953-7ae8-4b9e-a113-373d88615af2 None None] MFA Rules not processed for user `aae54952adc646969464f2c74d2ef61b`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:18.522146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0c92d953-7ae8-4b9e-a113-373d88615af2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:18.643539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 773/1542] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:18 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:18.656410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:18.658369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:18.762160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8fb273-df4f-4062-b7de-68635100f478 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:18.765933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c01d131-ae38-40de-a1fb-5c8790c1084c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:18.766978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c01d131-ae38-40de-a1fb-5c8790c1084c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:18.767266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c01d131-ae38-40de-a1fb-5c8790c1084c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:18.767541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c01d131-ae38-40de-a1fb-5c8790c1084c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:18.777351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c01d131-ae38-40de-a1fb-5c8790c1084c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:18.786658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c01d131-ae38-40de-a1fb-5c8790c1084c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:18.789549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 770/1543] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:18 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:20.067509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c92d953-7ae8-4b9e-a113-373d88615af2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:20.077875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:20.153885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c92d953-7ae8-4b9e-a113-373d88615af2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:20.159602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:20.160877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:20.161167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:20.161495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] PATH_INFO: `/v3/users/8883e606252a4cc09084e57fa871597b` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:20.163253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8883e606252a4cc09084e57fa871597b)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:20.165341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:20.349718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8883e606252a4cc09084e57fa871597b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.350544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8883e606252a4cc09084e57fa871597b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.364304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8883e606252a4cc09084e57fa871597b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.371518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8883e606252a4cc09084e57fa871597b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.372285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8883e606252a4cc09084e57fa871597b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.459971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 774/1544] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:20 2022] DELETE /identity/v3/users/8883e606252a4cc09084e57fa871597b => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:20.464378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:20.464841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:20.464928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:20.490834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:20.492141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:20.505455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:20.561915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 771/1545] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:20.571446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:20.575451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:20.630580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:20.632246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:20.645725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e0d45-7630-4975-b8e1-bfe35432325a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:20.646219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03de8720-2343-412b-bf54-b65fc44d3832 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:20.650049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84df57a2-d056-46c1-a77a-3b7b46a135a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:20.650165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df57a2-d056-46c1-a77a-3b7b46a135a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:20.650361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df57a2-d056-46c1-a77a-3b7b46a135a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:20.650630 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84df57a2-d056-46c1-a77a-3b7b46a135a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:20.651176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:20.651756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:20.652045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:20.652325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] PATH_INFO: `/v3/users/5969d9b178774b4ca799723b7bc78867` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:20.654006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5969d9b178774b4ca799723b7bc78867)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:20.656005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:20.660660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84df57a2-d056-46c1-a77a-3b7b46a135a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:20.662130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84df57a2-d056-46c1-a77a-3b7b46a135a9 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:20.672999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 775/1546] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:20.902708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5969d9b178774b4ca799723b7bc78867'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.920619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5969d9b178774b4ca799723b7bc78867'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.968602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5969d9b178774b4ca799723b7bc78867'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.970063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5969d9b178774b4ca799723b7bc78867'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:20.971304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5969d9b178774b4ca799723b7bc78867'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:21.024024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 772/1547] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:20 2022] DELETE /identity/v3/users/5969d9b178774b4ca799723b7bc78867 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:21.026747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:21.027050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:21.027339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:21.062160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:21.063767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:21.127423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:21.177599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 776/1548] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:21.189820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:21.192424 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:21.306907 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:21.310180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:21.327030 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f043d0d3-6910-490e-a84e-e2df8f5f5da5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:21.329768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9459e1e-efa5-48d9-b20f-ba9d5cd548ba None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:21.335478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:21.338713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:21.339039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:21.339329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:21.340229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:21.341526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:21.342161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:21.342896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] PATH_INFO: `/v3/projects/4582d981020b465daedbc2501e7ebe67` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:21.344639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4582d981020b465daedbc2501e7ebe67)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:21.347839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:21.366914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:21.370080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:21.402374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 773/1549] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:21.497112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4582d981020b465daedbc2501e7ebe67', 'request_id': 'req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:21.504618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4582d981020b465daedbc2501e7ebe67', 'request_id': 'req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:21.513041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4582d981020b465daedbc2501e7ebe67', 'request_id': 'req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:21.625461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] The token cache is being invalidate because project 4582d981020b465daedbc2501e7ebe67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:21.625819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:21.626416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:21.628260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 777/1550] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:21 2022] DELETE /identity/v3/projects/4582d981020b465daedbc2501e7ebe67 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:21.632192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:21.642721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd3fa80-b1ba-4021-a7f1-96cad5e6cd8b service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:21.686772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:21.726342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:21.726992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:21.727279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:21.727556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] PATH_INFO: `/v3/users/aae54952adc646969464f2c74d2ef61b` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:21.729215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aae54952adc646969464f2c74d2ef61b)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:21.731456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:22.004287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aae54952adc646969464f2c74d2ef61b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.017255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aae54952adc646969464f2c74d2ef61b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.088271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:22.089929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:22.104338 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aae54952adc646969464f2c74d2ef61b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.105051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aae54952adc646969464f2c74d2ef61b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.105952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81ec4f57-d70f-46d1-9a91-3c6cf0746186 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aae54952adc646969464f2c74d2ef61b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.119523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9764ffe1-b62b-4182-bfa2-b5f6e8ae3190 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:22.124380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:22.125091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.125397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.125678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] PATH_INFO: `/v3/projects/643e3664a27c4eafb7fea2c0ee9d8f81` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.126841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=643e3664a27c4eafb7fea2c0ee9d8f81)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:22.128751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 774/1551] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:21 2022] DELETE /identity/v3/users/aae54952adc646969464f2c74d2ef61b => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:22.129040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:22.316711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '643e3664a27c4eafb7fea2c0ee9d8f81', 'request_id': 'req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.322341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '643e3664a27c4eafb7fea2c0ee9d8f81', 'request_id': 'req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.330107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '643e3664a27c4eafb7fea2c0ee9d8f81', 'request_id': 'req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.429723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] The token cache is being invalidate because project 643e3664a27c4eafb7fea2c0ee9d8f81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:22.430018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.434113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-68336cc1-6f01-4c6b-aac9-c524e6c94f1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.434113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 778/1552] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:22 2022] DELETE /identity/v3/projects/643e3664a27c4eafb7fea2c0ee9d8f81 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:22.437632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080d3c40-c2bb-4f75-8ea2-ed36c4e989f3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.437920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080d3c40-c2bb-4f75-8ea2-ed36c4e989f3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.438246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080d3c40-c2bb-4f75-8ea2-ed36c4e989f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.473026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-080d3c40-c2bb-4f75-8ea2-ed36c4e989f3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:22.474591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-080d3c40-c2bb-4f75-8ea2-ed36c4e989f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:22.535351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 775/1553] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:22.543335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.543671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.543949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.563963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:22.565172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:22.573303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 779/1554] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:22.578332 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080d3c40-c2bb-4f75-8ea2-ed36c4e989f3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:22.588341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080d3c40-c2bb-4f75-8ea2-ed36c4e989f3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:22.594419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:22.599714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.599714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.599714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.599714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:22.607251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:22.608998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:22.628270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:22.630592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:22.633068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 776/1555] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:22.634008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:22.653032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee21ca2-d5f5-4244-8bb4-54ff351d9048 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:22.657923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1433f790-be6b-441f-9eec-db951b57b7a3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:22.668140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:22.668280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:22.668954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.669066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.669243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.669588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.669675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.669877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] PATH_INFO: `/v3/projects/5a61d3b9dce040d48d21149b9e48e159` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.671454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a61d3b9dce040d48d21149b9e48e159)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:22.671773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:22.673411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:22.678372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:22.710715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:22.783644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 777/1556] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:22 2022] POST /identity/v3/projects => generated 377 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:22.784768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:22.794516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a61d3b9dce040d48d21149b9e48e159', 'request_id': 'req-13361d65-8fce-4c60-bc9e-c11fab051f89'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.803682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a61d3b9dce040d48d21149b9e48e159', 'request_id': 'req-13361d65-8fce-4c60-bc9e-c11fab051f89'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.814630 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a61d3b9dce040d48d21149b9e48e159', 'request_id': 'req-13361d65-8fce-4c60-bc9e-c11fab051f89'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.819894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:22.867847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0993f970-fd94-44eb-a2d0-3dd30d467bed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:22.873633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:22.874518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.874804 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.875080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.875716 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:22.877676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:22.947734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] The token cache is being invalidate because project 5a61d3b9dce040d48d21149b9e48e159 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:22.947734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.948713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-13361d65-8fce-4c60-bc9e-c11fab051f89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:22.953564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 780/1557] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:22 2022] DELETE /identity/v3/projects/5a61d3b9dce040d48d21149b9e48e159 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:22.958263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f694a8ee-d17f-47fb-b80a-35566bc81443 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:22.959002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f694a8ee-d17f-47fb-b80a-35566bc81443 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:22.959425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f694a8ee-d17f-47fb-b80a-35566bc81443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:22.974791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 778/1558] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:22 2022] POST /identity/v3/users => generated 289 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:22.997889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.000562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:23.005202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f694a8ee-d17f-47fb-b80a-35566bc81443 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:23.006615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f694a8ee-d17f-47fb-b80a-35566bc81443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:23.029867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 781/1559] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:23.038203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b412ce-06f4-48de-b75a-065fbf2e82f5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.038447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b412ce-06f4-48de-b75a-065fbf2e82f5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.039024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b412ce-06f4-48de-b75a-065fbf2e82f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.048600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a6edc4-7958-4971-a947-18e8e39a2985 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.061173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.061173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.061173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.061173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.061530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.061530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.077362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 779/1560] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:22 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:23.081262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.083113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-28b412ce-06f4-48de-b75a-065fbf2e82f5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:23.083935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-28b412ce-06f4-48de-b75a-065fbf2e82f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:23.112500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 782/1561] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:23.112500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b412ce-06f4-48de-b75a-065fbf2e82f5 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.119565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08451f18-d17e-45db-9c0d-70d5692d9c88 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.131200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.132401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.134832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.135283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.136490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.139981 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.141359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b412ce-06f4-48de-b75a-065fbf2e82f5 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.151609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.152296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.152741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.153748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.154476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:23.154787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.156424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.164498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 780/1562] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:23 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:23.167822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.184104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 783/1563] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:23.189026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.199883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b463add-32ce-41fb-9daa-64e159e9d8be admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.201938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27c89a1-4ad3-449d-959f-1615cf46584e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.209353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80004ba-985b-4c99-82ef-fa4b2dc409ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.210060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80004ba-985b-4c99-82ef-fa4b2dc409ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.210347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80004ba-985b-4c99-82ef-fa4b2dc409ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.210731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80004ba-985b-4c99-82ef-fa4b2dc409ca admin admin] PATH_INFO: `/v3/projects/62895f66a6414c40b35e83680c1809eb/users/797382c3e1dc454faed3eba66ec1196c/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.225854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.226501 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.228045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.228045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.228045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.229986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.264756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80004ba-985b-4c99-82ef-fa4b2dc409ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=62895f66a6414c40b35e83680c1809eb, user_id=797382c3e1dc454faed3eba66ec1196c, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.267600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80004ba-985b-4c99-82ef-fa4b2dc409ca admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.386524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 781/1564] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:23 2022] PUT /identity/v3/projects/62895f66a6414c40b35e83680c1809eb/users/797382c3e1dc454faed3eba66ec1196c/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:23.399776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa749fdc-1e1e-48d9-80ec-9754b64a7d04 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.400277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa749fdc-1e1e-48d9-80ec-9754b64a7d04 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.400614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa749fdc-1e1e-48d9-80ec-9754b64a7d04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.430078 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fa749fdc-1e1e-48d9-80ec-9754b64a7d04 None None] MFA Rules not processed for user `797382c3e1dc454faed3eba66ec1196c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:23.431291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fa749fdc-1e1e-48d9-80ec-9754b64a7d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:23.497441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 784/1565] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:23 2022] POST /identity/v3/projects => generated 383 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:23.498978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 782/1566] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:23.501512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.503330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa749fdc-1e1e-48d9-80ec-9754b64a7d04 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.573636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a595e-e0fd-4886-95ce-0c6c82d2c3ee admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.579395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:23.580941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.581852 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.582138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.582509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.583279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.585527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.592562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa749fdc-1e1e-48d9-80ec-9754b64a7d04 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.598315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.599590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.600318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.600707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.601525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.603643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.693669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 785/1567] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:23 2022] POST /identity/v3/users => generated 292 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:23.697775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.710057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af001202-4105-46cd-915f-0532e2cf1fef admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.716237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 783/1568] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:23 2022] POST /identity/v3/projects => generated 373 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:23.716705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.716705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.716705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.716705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.720009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.720009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.721579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.729415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 786/1569] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:23.734116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.744180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ef8e0-f07f-4d3c-a328-3d5ddb1c71c7 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.791586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=6,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:23.799924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b39d03-5302-4b44-9909-b39168f18a10 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.806184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c8f418-ad8a-4d54-addb-e50a3cfbfebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.807200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c8f418-ad8a-4d54-addb-e50a3cfbfebb admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.807586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c8f418-ad8a-4d54-addb-e50a3cfbfebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.807964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c8f418-ad8a-4d54-addb-e50a3cfbfebb admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.808712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c8f418-ad8a-4d54-addb-e50a3cfbfebb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.812055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.812213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c8f418-ad8a-4d54-addb-e50a3cfbfebb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.812704 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.812993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.813270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.814001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.815791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:23.828358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 787/1570] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:23 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:23.832583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:23.843863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282fba48-1716-49f0-935d-47a6c6e90f89 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:23.849753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:23.850424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:23.850706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:23.850984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] PATH_INFO: `/v3/projects/a9441acf0d244ed584162653053f8168/users/0f1a326e6644434bb651f4f791852ae5/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:23.871761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9441acf0d244ed584162653053f8168, user_id=0f1a326e6644434bb651f4f791852ae5, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:23.879018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.035893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 788/1571] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:23 2022] PUT /identity/v3/projects/a9441acf0d244ed584162653053f8168/users/0f1a326e6644434bb651f4f791852ae5/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:24.037340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.037550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 784/1572] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:23 2022] POST /identity/v3/users => generated 287 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:24.049551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08df71d3-b780-4539-b205-3006e09a6a44 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.049824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08df71d3-b780-4539-b205-3006e09a6a44 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.050105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08df71d3-b780-4539-b205-3006e09a6a44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.102144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-08df71d3-b780-4539-b205-3006e09a6a44 None None] MFA Rules not processed for user `0f1a326e6644434bb651f4f791852ae5`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:24.103387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-08df71d3-b780-4539-b205-3006e09a6a44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:24.160622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed6f6c-a624-47da-86fa-a5d7609eba60 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.182919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.188180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.188180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.188180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.188180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.188180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.201462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 789/1573] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:24 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:24.205641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.209677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 785/1574] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:24.216001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08df71d3-b780-4539-b205-3006e09a6a44 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.218673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec365a-c71e-4bf3-b602-155e8cc7998c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.224336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.225844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.226239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.226615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.227778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.230354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.231982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08df71d3-b780-4539-b205-3006e09a6a44 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.244773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 790/1575] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:24 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:24.250140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.250785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.251074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.251358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.251474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.255688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.260877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.265046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4529e-9f24-4487-93c6-08d32520a329 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.276126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175acd96-6951-4c62-aba1-763f1f45844d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.276964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175acd96-6951-4c62-aba1-763f1f45844d admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.277371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175acd96-6951-4c62-aba1-763f1f45844d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.277783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175acd96-6951-4c62-aba1-763f1f45844d admin admin] PATH_INFO: `/v3/projects/539bf4ccc56446da9be4008e472e307a/users/d1f97989d05b412e94009df6f3a0c485/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.299428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175acd96-6951-4c62-aba1-763f1f45844d admin admin] RBAC: Authorizing `identity:create_grant(project_id=539bf4ccc56446da9be4008e472e307a, user_id=d1f97989d05b412e94009df6f3a0c485, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.302266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175acd96-6951-4c62-aba1-763f1f45844d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.393790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 786/1576] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:24 2022] POST /identity/v3/projects => generated 383 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:24.396434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.404376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 791/1577] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:24 2022] PUT /identity/v3/projects/539bf4ccc56446da9be4008e472e307a/users/d1f97989d05b412e94009df6f3a0c485/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:24.416914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555db7e8-ffb2-4095-ac7a-ec478f28bd3e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.416914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555db7e8-ffb2-4095-ac7a-ec478f28bd3e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.417253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555db7e8-ffb2-4095-ac7a-ec478f28bd3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.449605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-555db7e8-ffb2-4095-ac7a-ec478f28bd3e None None] MFA Rules not processed for user `d1f97989d05b412e94009df6f3a0c485`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:24.451831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-555db7e8-ffb2-4095-ac7a-ec478f28bd3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:24.472752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:24.506322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feb67ca-cb0a-43c4-b409-ef3cc80cb01d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.565116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 792/1578] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:24.574232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555db7e8-ffb2-4095-ac7a-ec478f28bd3e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.588067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:24.597300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.599155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.599666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.599865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.600514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.602991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.675658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555db7e8-ffb2-4095-ac7a-ec478f28bd3e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.681871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.682397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.682690 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.683812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.684347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.686879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.733098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 787/1579] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:24 2022] POST /identity/v3/users => generated 298 bytes in 339 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:24.737016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.747413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3f8cfc-4960-49f1-9ede-f54688f201b7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.756051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.756681 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.759853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.759853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.759853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.759853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.775836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 788/1580] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:24 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:24.775836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.777888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:24.785907 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 793/1581] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:24 2022] POST /identity/v3/projects => generated 377 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:24.786768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:24.792392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72051e2-1e61-4f16-8dfb-f645a915f13b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.855252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b64541-b86a-4b6a-833e-190641a41afb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:24.861084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.862067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.862506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.862841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] PATH_INFO: `/v3/projects/31b2244e01424b1d9d793315e88d08d2/users/41a74558835b4890be132fe49063d773/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.863148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:24.864856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:24.865160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:24.865633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] PATH_INFO: `/v3/users/0763f168579b4e46b162240adcb48ba3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:24.867390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0763f168579b4e46b162240adcb48ba3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.870279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.881524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=31b2244e01424b1d9d793315e88d08d2, user_id=41a74558835b4890be132fe49063d773, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:24.884253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:24.986191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 794/1582] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:24 2022] PUT /identity/v3/projects/31b2244e01424b1d9d793315e88d08d2/users/41a74558835b4890be132fe49063d773/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:24.987520 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.051958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e57ceb-916d-4cf8-b9e3-f7897c9aeb77 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.056537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.057139 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.057452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.057725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.058345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.060806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.107452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0763f168579b4e46b162240adcb48ba3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:25.118407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:25.119712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0763f168579b4e46b162240adcb48ba3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:25.153551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 795/1583] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:24 2022] POST /identity/v3/users => generated 295 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:25.155209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.165221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e6a337-be8a-45ae-b862-6ca030117675 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.170913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.171527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.171793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.172118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.172754 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.174434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.188174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 796/1584] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:25.189887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.200791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eeddcd-6bf8-410f-a328-bc0981a5d9b7 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.205843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.206490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.207721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.208080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.208261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0763f168579b4e46b162240adcb48ba3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:25.209095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0763f168579b4e46b162240adcb48ba3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:25.209932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87e8076c-9879-4212-b837-cceaf4fba710 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0763f168579b4e46b162240adcb48ba3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:25.210184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.214341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.229089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 797/1586] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:25 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:25.229898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.235430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:25.235587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 789/1586] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:24 2022] DELETE /identity/v3/users/0763f168579b4e46b162240adcb48ba3 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:25.235587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49e63ff-e125-497f-a756-2ad3d92ecef9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.235587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49e63ff-e125-497f-a756-2ad3d92ecef9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.235587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49e63ff-e125-497f-a756-2ad3d92ecef9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.242859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:25.250767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c04204e-8b65-4967-ae25-a3a7cb2acc05 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.291545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c49e63ff-e125-497f-a756-2ad3d92ecef9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:25.292874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c49e63ff-e125-497f-a756-2ad3d92ecef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:25.319289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.320496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.320849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.321132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] PATH_INFO: `/v3/projects/31b2244e01424b1d9d793315e88d08d2/users/41a74558835b4890be132fe49063d773/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.323382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=31b2244e01424b1d9d793315e88d08d2, user_id=41a74558835b4890be132fe49063d773, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.326336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.344539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 798/1587] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:25 2022] PUT /identity/v3/projects/31b2244e01424b1d9d793315e88d08d2/users/41a74558835b4890be132fe49063d773/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:25.346392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.357776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:25.410494 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 790/1588] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:25.415153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.415397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.415687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.445158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03450e-1257-40dd-89e6-5ca21bc010b1 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.449308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] MFA Rules not processed for user `41a74558835b4890be132fe49063d773`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:25.451323 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:25.453179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.453959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.454288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.454623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] PATH_INFO: `/v3/projects/eea0d076a3ef493baa08f9cd77410e4c/users/494ecbada68c4178bfd050bb42e3dce4/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.476105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eea0d076a3ef493baa08f9cd77410e4c, user_id=494ecbada68c4178bfd050bb42e3dce4, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.479033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.544647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 791/1589] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:25 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:25.549936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.552229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:25.567879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 799/1590] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:25 2022] PUT /identity/v3/projects/eea0d076a3ef493baa08f9cd77410e4c/users/494ecbada68c4178bfd050bb42e3dce4/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:25.568818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.573094 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:25.633971 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76ba5f5-0663-4726-ad1d-77b0b9825979 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.656321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:25.695938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eaff1e-f281-46c0-9746-6dfe1a4fcc30 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.696779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.698388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.698674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.698950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.700724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.701648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.702138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.702613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.716413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:25.783904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.786283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.800545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 792/1591] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:25.803303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.804778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.822906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.839519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6cb78-8596-48e9-ab42-b55e4db01725 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.846623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 800/1592] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:25 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:25.855437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeed3a7e-bf54-45b6-8d59-20ed573cecad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.856688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeed3a7e-bf54-45b6-8d59-20ed573cecad admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.858703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeed3a7e-bf54-45b6-8d59-20ed573cecad admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.858703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeed3a7e-bf54-45b6-8d59-20ed573cecad admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.858703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeed3a7e-bf54-45b6-8d59-20ed573cecad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.861060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeed3a7e-bf54-45b6-8d59-20ed573cecad admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:25.895024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:25.895630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 793/1593] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:25 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:25.921030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ce77e-42f5-4914-8cd5-f09bad8eb561 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:25.952617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:25.958717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:25.959016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:25.962557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] PATH_INFO: `/v3/projects/eea0d076a3ef493baa08f9cd77410e4c/users/494ecbada68c4178bfd050bb42e3dce4/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:25.969764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eea0d076a3ef493baa08f9cd77410e4c, user_id=494ecbada68c4178bfd050bb42e3dce4, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:25.977676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:26.001035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 801/1594] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:25 2022] PUT /identity/v3/projects/eea0d076a3ef493baa08f9cd77410e4c/users/494ecbada68c4178bfd050bb42e3dce4/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:26.016847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:26.016847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:26.016847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:26.060426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] MFA Rules not processed for user `494ecbada68c4178bfd050bb42e3dce4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:26.061652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:26.084337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:26.179806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f85d186-f64b-4988-a71e-9a6c8bce8dd8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:26.186625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:26.187273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:26.187558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:26.188073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:26.190038 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:26.197944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 794/1595] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:26 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:26.276573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:26.278886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:26.289001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 802/1596] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:26.321602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:26.322830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:26.323263 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:26.333229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1957e8-20cc-4539-bc18-bf155d87c1d8 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:26.337685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:26.338314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:26.338600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:26.338878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:26.341304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:26.400836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:26.402617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:26.420285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:26.421908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:26.422536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd33cb-8ecb-454f-8147-f2724352c3da None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:26.427496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:26.428209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:26.428497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:26.429139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] PATH_INFO: `/v3/projects/72df80d266cf4e04915a5e7d2b556eaf` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:26.430536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72df80d266cf4e04915a5e7d2b556eaf)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:26.432420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:26.432913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 803/1597] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:26 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:26.620184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72df80d266cf4e04915a5e7d2b556eaf', 'request_id': 'req-13208155-a982-422a-aea4-705ebc5c5023'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:26.743021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72df80d266cf4e04915a5e7d2b556eaf', 'request_id': 'req-13208155-a982-422a-aea4-705ebc5c5023'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:26.755859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72df80d266cf4e04915a5e7d2b556eaf', 'request_id': 'req-13208155-a982-422a-aea4-705ebc5c5023'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:26.776026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] The token cache is being invalidate because project 72df80d266cf4e04915a5e7d2b556eaf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:26.777596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:26.777596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:26.779745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 795/1598] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:26 2022] DELETE /identity/v3/projects/72df80d266cf4e04915a5e7d2b556eaf => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:26.784510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:26.811057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70872bde-715a-4b71-ad0c-d0212e1f87f4 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:26.910639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:26.911270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:26.911551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:26.911825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] PATH_INFO: `/v3/users/7f9d85665b7e4e2e94ac3e5f68f0dbbe` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:26.912928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f9d85665b7e4e2e94ac3e5f68f0dbbe)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:26.914877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:27.125056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f9d85665b7e4e2e94ac3e5f68f0dbbe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.126294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f9d85665b7e4e2e94ac3e5f68f0dbbe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.150501 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f9d85665b7e4e2e94ac3e5f68f0dbbe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.163475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f9d85665b7e4e2e94ac3e5f68f0dbbe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.164266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e607e8de-a100-4260-975f-957a50c1073b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f9d85665b7e4e2e94ac3e5f68f0dbbe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.165374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:27.168573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:27.197665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13208155-a982-422a-aea4-705ebc5c5023 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:27.203653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:27.204325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:27.204609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:27.204889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] PATH_INFO: `/v3/users/0f1a326e6644434bb651f4f791852ae5` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:27.206483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f1a326e6644434bb651f4f791852ae5)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:27.208839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:27.266201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 804/1599] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:26 2022] DELETE /identity/v3/users/7f9d85665b7e4e2e94ac3e5f68f0dbbe => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:27.452316 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f1a326e6644434bb651f4f791852ae5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.468758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f1a326e6644434bb651f4f791852ae5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.550112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f1a326e6644434bb651f4f791852ae5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.550761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f1a326e6644434bb651f4f791852ae5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.551481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f1a326e6644434bb651f4f791852ae5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:27.566669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 796/1600] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:27 2022] DELETE /identity/v3/users/0f1a326e6644434bb651f4f791852ae5 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:27.571216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:27.571663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:27.572058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:27.605412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:27.608215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:27.704103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 805/1601] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:27.710890 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:27.712318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:27.773519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:27.777633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f61c5-d7d7-44a6-8e00-8945664ac9f8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:27.781275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39fe4ee-6260-4ae7-ad68-893170c3db0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:27.781968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39fe4ee-6260-4ae7-ad68-893170c3db0b service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:27.782252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39fe4ee-6260-4ae7-ad68-893170c3db0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:27.782527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39fe4ee-6260-4ae7-ad68-893170c3db0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:27.792254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39fe4ee-6260-4ae7-ad68-893170c3db0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:27.793901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39fe4ee-6260-4ae7-ad68-893170c3db0b service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:27.803617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 797/1602] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:27.862826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:27.866015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:27.899636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69eda191-ad7b-40b1-8f2f-5805c0fa8916 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:27.928271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:27.928936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:27.929222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:27.929517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] PATH_INFO: `/v3/projects/4a655235a6b542228be1db9ddaf4ad7f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:27.932985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a655235a6b542228be1db9ddaf4ad7f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:27.935383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:27.937064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:28.092732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a655235a6b542228be1db9ddaf4ad7f', 'request_id': 'req-8d7b3888-e250-45a8-9865-7f482e6148cd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.099593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a655235a6b542228be1db9ddaf4ad7f', 'request_id': 'req-8d7b3888-e250-45a8-9865-7f482e6148cd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.108861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a655235a6b542228be1db9ddaf4ad7f', 'request_id': 'req-8d7b3888-e250-45a8-9865-7f482e6148cd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.173342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] The token cache is being invalidate because project 4a655235a6b542228be1db9ddaf4ad7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:28.173638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.181619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8d7b3888-e250-45a8-9865-7f482e6148cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.187427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 806/1603] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:27 2022] DELETE /identity/v3/projects/4a655235a6b542228be1db9ddaf4ad7f => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:28.188940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e90d9c1-0d0c-4654-80e8-0449f1d3244e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.189403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e90d9c1-0d0c-4654-80e8-0449f1d3244e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.189588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e90d9c1-0d0c-4654-80e8-0449f1d3244e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.214577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1e90d9c1-0d0c-4654-80e8-0449f1d3244e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:28.216583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1e90d9c1-0d0c-4654-80e8-0449f1d3244e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:28.367717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 798/1604] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:28.383248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd88d7-9b84-4410-8d72-f74b8e70d1e7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.383772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd88d7-9b84-4410-8d72-f74b8e70d1e7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.383772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd88d7-9b84-4410-8d72-f74b8e70d1e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.406520 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-1ffd88d7-9b84-4410-8d72-f74b8e70d1e7 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:28.407761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1ffd88d7-9b84-4410-8d72-f74b8e70d1e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:28.416499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 807/1605] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:28.421523 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e90d9c1-0d0c-4654-80e8-0449f1d3244e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:28.431448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e90d9c1-0d0c-4654-80e8-0449f1d3244e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:28.437123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:28.437776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.439013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.439013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.439013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:28.439292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:28.440856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:28.453782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 799/1606] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:28.458771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffd88d7-9b84-4410-8d72-f74b8e70d1e7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:28.472271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffd88d7-9b84-4410-8d72-f74b8e70d1e7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:28.482544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:28.482544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.482544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.482544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.482874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:28.482874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:28.508373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:28.510666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:28.545330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d446e324-d89c-4fd1-8481-bc095b6817b6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:28.551635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:28.552814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.553009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.553286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] PATH_INFO: `/v3/projects/a9441acf0d244ed584162653053f8168` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.554550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9441acf0d244ed584162653053f8168)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:28.557252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:28.608011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 808/1607] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:28 2022] POST /identity/v3/projects => generated 365 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:28.612555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:28.711750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9441acf0d244ed584162653053f8168', 'request_id': 'req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.715822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd18aa6d-7ccb-4608-a248-8decf2072e9f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:28.830740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9441acf0d244ed584162653053f8168', 'request_id': 'req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.843000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9441acf0d244ed584162653053f8168', 'request_id': 'req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.859911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:28.860360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] The token cache is being invalidate because project a9441acf0d244ed584162653053f8168 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:28.860668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.861187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.861525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.861800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.862208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:28.862493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:28.865828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:28.870101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 800/1608] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:28 2022] DELETE /identity/v3/projects/a9441acf0d244ed584162653053f8168 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:28.870101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:28.870101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:28.952173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfc4c3b-5497-43a6-9956-4a80fd99a3e5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:28.957479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:28.957867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:28.958171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.958610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.958896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.961005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:28.966326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 809/1609] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:28 2022] POST /identity/v3/users => generated 283 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:28.974867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:28.974867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:28.976974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:28.987250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabfd8ce-9995-4c7e-bf86-6fa3d1f95c59 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:28.993840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:28.994545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:28.994875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:28.995340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] PATH_INFO: `/v3/users/41a74558835b4890be132fe49063d773` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:28.997511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] RBAC: Authorizing `identity:delete_user(user_id=41a74558835b4890be132fe49063d773)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:29.001980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:29.046734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:29.048693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:29.062430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 801/1610] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:29.062430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:29.062430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:29.079113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d379f4b-439b-4567-a4dd-42283b1071c3 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:29.085145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:29.086004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:29.086609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:29.086971 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:29.087722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:29.089452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:29.102324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 802/1611] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:29 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:29.106296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:29.120873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c350fa-c3a5-4f8b-85cd-d5d88bf9056f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:29.128453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:29.129528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:29.129931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:29.130218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:29.131133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:29.133221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:29.153470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 803/1612] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:29 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:29.155339 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:29.169120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1e33f0-c593-45b1-acd9-1f53b3ba20cb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:29.175191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7d19e2-e08d-47e4-91d2-664fa54cd642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:29.175634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7d19e2-e08d-47e4-91d2-664fa54cd642 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:29.175921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7d19e2-e08d-47e4-91d2-664fa54cd642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:29.176204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7d19e2-e08d-47e4-91d2-664fa54cd642 admin admin] PATH_INFO: `/v3/projects/fd351d224f27454e96502e67e9a89873/users/fce4e45562744a6d98acd0addfdf1b18/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:29.194697 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7d19e2-e08d-47e4-91d2-664fa54cd642 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd351d224f27454e96502e67e9a89873, user_id=fce4e45562744a6d98acd0addfdf1b18, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:29.202399 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7d19e2-e08d-47e4-91d2-664fa54cd642 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:29.213463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41a74558835b4890be132fe49063d773'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:29.214250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41a74558835b4890be132fe49063d773'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:29.230246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41a74558835b4890be132fe49063d773'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:29.238743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41a74558835b4890be132fe49063d773'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:29.239911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41a74558835b4890be132fe49063d773'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:29.347894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 804/1614] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:29 2022] PUT /identity/v3/projects/fd351d224f27454e96502e67e9a89873/users/fce4e45562744a6d98acd0addfdf1b18/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:29.348214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 810/1614] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:28 2022] DELETE /identity/v3/users/41a74558835b4890be132fe49063d773 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:29.350751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bfc3e-738f-46f2-85e9-bbdecd17afa5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:29.351190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bfc3e-738f-46f2-85e9-bbdecd17afa5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:29.351602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bfc3e-738f-46f2-85e9-bbdecd17afa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:29.420479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9c6bfc3e-738f-46f2-85e9-bbdecd17afa5 None None] MFA Rules not processed for user `fce4e45562744a6d98acd0addfdf1b18`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:29.423780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9c6bfc3e-738f-46f2-85e9-bbdecd17afa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:29.525106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 805/1615] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:29 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:29.526386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:29.528799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:29.630517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28161ecc-9ca5-4f44-aa76-24907e66333f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:29.636306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:29.637762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:29.638398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:29.638868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:29.639745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:29.642269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:29.773932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 811/1616] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:29 2022] POST /identity/v3/projects => generated 365 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:29.778495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6bfc3e-738f-46f2-85e9-bbdecd17afa5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:29.825261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6bfc3e-738f-46f2-85e9-bbdecd17afa5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:29.922951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:29.924006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:29.924117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:29.924317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:29.924986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:29.927594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:29.970632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:29.972736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:29.997921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6527eaa9-003a-4884-b129-6fc8cbb3aa6c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.007302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.007968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.008249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.008560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] PATH_INFO: `/v3/projects/31b2244e01424b1d9d793315e88d08d2` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.009708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=31b2244e01424b1d9d793315e88d08d2)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.011673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.047917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 806/1617] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:29 2022] POST /identity/v3/users => generated 289 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:30.051879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.064166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2eceb0-2dd7-4684-a6dc-03ca8999655f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.069846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.070479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.070807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.071083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.072486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.074316 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.086136 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 807/1618] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:30 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:30.090043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.112463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:30.119254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525edb3-e31b-4300-a529-b12bff2a2766 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.133616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.134430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.134707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.134995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] PATH_INFO: `/v3/projects/36cdb00e138e4fdd9c6cfdfd28ddcfec/users/5f4ebc42a8364c6bb4ab5f30a01a5025/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.157537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31b2244e01424b1d9d793315e88d08d2', 'request_id': 'req-ea217d47-07df-48c8-baf1-cf3c178a4ea7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:30.158916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:30.168961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] RBAC: Authorizing `identity:create_grant(project_id=36cdb00e138e4fdd9c6cfdfd28ddcfec, user_id=5f4ebc42a8364c6bb4ab5f30a01a5025, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.170507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31b2244e01424b1d9d793315e88d08d2', 'request_id': 'req-ea217d47-07df-48c8-baf1-cf3c178a4ea7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:30.172250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.179803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31b2244e01424b1d9d793315e88d08d2', 'request_id': 'req-ea217d47-07df-48c8-baf1-cf3c178a4ea7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:30.275161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 808/1619] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:30 2022] PUT /identity/v3/projects/36cdb00e138e4fdd9c6cfdfd28ddcfec/users/5f4ebc42a8364c6bb4ab5f30a01a5025/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:30.276962 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.277080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] The token cache is being invalidate because project 31b2244e01424b1d9d793315e88d08d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:30.277667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:30.278801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea217d47-07df-48c8-baf1-cf3c178a4ea7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:30.285722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 812/1620] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:29 2022] DELETE /identity/v3/projects/31b2244e01424b1d9d793315e88d08d2 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:30.288788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868d53b0-0c3e-473b-ae01-9ec8bbae07b7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.289242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868d53b0-0c3e-473b-ae01-9ec8bbae07b7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.292171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868d53b0-0c3e-473b-ae01-9ec8bbae07b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.319857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-868d53b0-0c3e-473b-ae01-9ec8bbae07b7 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:30.322364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-868d53b0-0c3e-473b-ae01-9ec8bbae07b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:30.410303 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ada510-887a-411a-83a1-cd4b2da3728c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.423660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.424720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.425189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.425692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.426490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.430719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.439918 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 813/1621] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:30.449294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c6bbe-3e0e-45d2-ae14-28493bbdd7bf None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.450497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c6bbe-3e0e-45d2-ae14-28493bbdd7bf None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.450783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c6bbe-3e0e-45d2-ae14-28493bbdd7bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.471620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 809/1622] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:30 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:30.475867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.480083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:30.482707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-056c6bbe-3e0e-45d2-ae14-28493bbdd7bf None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:30.484319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-056c6bbe-3e0e-45d2-ae14-28493bbdd7bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:30.503216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 814/1623] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:30.504665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c6bbe-3e0e-45d2-ae14-28493bbdd7bf None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.507341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0020056-3f70-417d-8785-6331eb806cf8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.517601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c6bbe-3e0e-45d2-ae14-28493bbdd7bf None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.522180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95b6674-6915-4827-9482-009d8421fe4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.522997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95b6674-6915-4827-9482-009d8421fe4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.523353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95b6674-6915-4827-9482-009d8421fe4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.523700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95b6674-6915-4827-9482-009d8421fe4b admin admin] PATH_INFO: `/v3/projects/36cdb00e138e4fdd9c6cfdfd28ddcfec/users/5f4ebc42a8364c6bb4ab5f30a01a5025/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.528562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.528562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.528562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.528562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.529460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:30.529460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.530245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.531489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95b6674-6915-4827-9482-009d8421fe4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=36cdb00e138e4fdd9c6cfdfd28ddcfec, user_id=5f4ebc42a8364c6bb4ab5f30a01a5025, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.534484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95b6674-6915-4827-9482-009d8421fe4b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.551565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 815/1624] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:30.556487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.561197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 810/1625] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:30 2022] PUT /identity/v3/projects/36cdb00e138e4fdd9c6cfdfd28ddcfec/users/5f4ebc42a8364c6bb4ab5f30a01a5025/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:30.571605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.572094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.572478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.596201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5a1910-28d5-49c5-8d8a-5c5592535706 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.617789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] MFA Rules not processed for user `5f4ebc42a8364c6bb4ab5f30a01a5025`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:30.619317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:30.673734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.674030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.674438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.675384 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.676062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.678308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.707026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 811/1626] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:30 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:30.719711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.721537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:30.786724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0076b1-5f41-46be-a4a4-e4630a0dd912 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.790886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 816/1627] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:30 2022] POST /identity/v3/projects => generated 385 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:30.792048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:30.874549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.875783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.876135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.876530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.958288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d711d65-e85b-462b-9f7c-3bfff87e3e1c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:30.961909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.963451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.974674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:30.975446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:30.975917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:30.976289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:30.977038 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:30.981804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:30.982205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 812/1628] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:30 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:31.050112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:31.052076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:31.063790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37427c89-f7ca-4348-b25c-96e44f02c12b service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:31.068037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:31.069007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.073408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.073408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.073408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:31.114950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 817/1629] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:30 2022] POST /identity/v3/users => generated 293 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:31.123608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:31.165562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6c420-b9aa-4123-af41-08da2950a2d0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:31.167658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:31.172919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:31.177894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:31.181549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.181876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.182162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.183097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:31.194503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 813/1630] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:31 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:31.208544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:31.231457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 818/1631] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:31 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:31.235154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:31.252335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:31.253362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea587be-16ed-4f8e-947c-45db5c4bfa20 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:31.259154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0b3fd1-25a7-421a-84f9-87599f602dc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:31.260154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b3fd1-25a7-421a-84f9-87599f602dc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.260613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b3fd1-25a7-421a-84f9-87599f602dc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.261057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b3fd1-25a7-421a-84f9-87599f602dc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.262720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0b3fd1-25a7-421a-84f9-87599f602dc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:31.264545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0b3fd1-25a7-421a-84f9-87599f602dc5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:31.281406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 814/1632] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:31 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:31.283976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:31.300683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850aaba1-3d75-4fcf-bbf6-1c98494a4a5e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:31.306224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:31.307562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.308028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.308478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] PATH_INFO: `/v3/projects/089c3548781b4dddac6796d8da77b562/users/2c8d0df59a54449485d86b8533496d1f/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.335847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] RBAC: Authorizing `identity:create_grant(project_id=089c3548781b4dddac6796d8da77b562, user_id=2c8d0df59a54449485d86b8533496d1f, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:31.339498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:31.420170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 819/1633] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:31 2022] PUT /identity/v3/projects/089c3548781b4dddac6796d8da77b562/users/2c8d0df59a54449485d86b8533496d1f/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:31.435967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba0c664-a1cf-46f3-827d-50bd41e9b33d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.436418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba0c664-a1cf-46f3-827d-50bd41e9b33d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.436872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba0c664-a1cf-46f3-827d-50bd41e9b33d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.469751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fba0c664-a1cf-46f3-827d-50bd41e9b33d None None] MFA Rules not processed for user `2c8d0df59a54449485d86b8533496d1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:31.472200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fba0c664-a1cf-46f3-827d-50bd41e9b33d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:31.476740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:31.535828 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6713e2b-88b9-46f1-832e-635fba2dc89a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:31.545019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:31.545264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.545614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.545895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.547783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:31.563600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 815/1634] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:31.565021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba0c664-a1cf-46f3-827d-50bd41e9b33d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:31.649504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:31.651021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:31.660304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba0c664-a1cf-46f3-827d-50bd41e9b33d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:31.664462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 820/1635] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:31 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:31.668233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:31.673450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.673450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.673450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.673450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:31.673997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:31.734273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:31.775973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 816/1636] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:31 2022] POST /identity/v3/projects => generated 383 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:31.780548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:31.795530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d76d5-d891-4a7a-8af1-22ecdc8a82ba service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:31.873121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:31.873810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:31.874097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:31.874377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:31.875029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:31.876957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:32.007800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 821/1637] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:31 2022] POST /identity/v3/users => generated 292 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:32.011643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:32.022862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0c88c8-3c93-425d-9add-1414d56e894a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:32.041099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:32.041814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.042098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.043299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.044464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:32.046435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:32.059910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 817/1638] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:32 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:32.063858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:32.075834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8172fd-a987-4fa6-a21a-fe794935d6cd admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:32.080538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43179662-7b4b-42df-b0dd-b099e425ea3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:32.081148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43179662-7b4b-42df-b0dd-b099e425ea3b admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.081542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43179662-7b4b-42df-b0dd-b099e425ea3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.082099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43179662-7b4b-42df-b0dd-b099e425ea3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.086326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43179662-7b4b-42df-b0dd-b099e425ea3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:32.086326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43179662-7b4b-42df-b0dd-b099e425ea3b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:32.095417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 822/1639] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:32.099942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:32.113052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f810785-42d3-4992-b54b-97ae2b672b27 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:32.118253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:32.119104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.119456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.119790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] PATH_INFO: `/v3/projects/be99d0e1d4904f8885593c32f2e0b839/users/aee73d6b32194d1487abf1e309af9dcd/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.136219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=be99d0e1d4904f8885593c32f2e0b839, user_id=aee73d6b32194d1487abf1e309af9dcd, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:32.139738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:32.218770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 818/1640] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:32 2022] PUT /identity/v3/projects/be99d0e1d4904f8885593c32f2e0b839/users/aee73d6b32194d1487abf1e309af9dcd/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:32.220587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c4cd07-029a-4912-9bcd-32b3154183d8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.220878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c4cd07-029a-4912-9bcd-32b3154183d8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.221137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c4cd07-029a-4912-9bcd-32b3154183d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.224900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:32.255989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-24c4cd07-029a-4912-9bcd-32b3154183d8 None None] MFA Rules not processed for user `aee73d6b32194d1487abf1e309af9dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:32.261983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-24c4cd07-029a-4912-9bcd-32b3154183d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:32.388599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 823/1641] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:32.393932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:32.521822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01292019-939e-4240-ac3e-5e8fdf3a3f6d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:32.528847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:32.530220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.530509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.531110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.531807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:32.536011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:32.640092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 819/1642] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:32 2022] POST /identity/v3/projects => generated 383 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:32.644689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c4cd07-029a-4912-9bcd-32b3154183d8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:32.670012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c4cd07-029a-4912-9bcd-32b3154183d8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:32.744866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:32.748602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:32.749220 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.749600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.757800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.757800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:32.757800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:32.878834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 824/1643] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:32 2022] POST /identity/v3/users => generated 298 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:32.882533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:32.896093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f447f695-5932-4479-8d48-0d2b45f16e6d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:32.902301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:32.902979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.903380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.903712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.904318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:32.906175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:32.923801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 820/1644] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:32 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:32.925361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:32.937757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aed5c7e-58a6-4e30-8801-f6410a73bf10 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:32.942700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:32.943509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:32.944007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:32.944463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] PATH_INFO: `/v3/projects/e00d29c292fe43b7abc29883ca18cdbe/users/2adcb2d5a54d4ceaa65a63c7bc03f44f/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:32.964843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e00d29c292fe43b7abc29883ca18cdbe, user_id=2adcb2d5a54d4ceaa65a63c7bc03f44f, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:32.968523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:33.013457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 825/1645] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:32 2022] PUT /identity/v3/projects/e00d29c292fe43b7abc29883ca18cdbe/users/2adcb2d5a54d4ceaa65a63c7bc03f44f/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:33.017572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:33.030204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61ec3b-f9d1-40b4-83ff-ba582ad365b0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:33.091839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabed760-aaab-4b9b-b5d6-369ba97994ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:33.092471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabed760-aaab-4b9b-b5d6-369ba97994ab admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:33.092761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabed760-aaab-4b9b-b5d6-369ba97994ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:33.093042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabed760-aaab-4b9b-b5d6-369ba97994ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:33.093726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabed760-aaab-4b9b-b5d6-369ba97994ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:33.095396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabed760-aaab-4b9b-b5d6-369ba97994ab admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:33.105862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 821/1646] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:33 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:33.110494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:33.120777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f74b188-8e44-421a-b623-fb7849434da1 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:33.126090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:33.126734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:33.127011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:33.127284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] PATH_INFO: `/v3/projects/e00d29c292fe43b7abc29883ca18cdbe/users/2adcb2d5a54d4ceaa65a63c7bc03f44f/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:33.129147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e00d29c292fe43b7abc29883ca18cdbe, user_id=2adcb2d5a54d4ceaa65a63c7bc03f44f, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:33.131799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:33.142519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 826/1647] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:33 2022] PUT /identity/v3/projects/e00d29c292fe43b7abc29883ca18cdbe/users/2adcb2d5a54d4ceaa65a63c7bc03f44f/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:33.152723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7221ec4c-e992-4ae6-8614-b4755d9986f7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:33.153336 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7221ec4c-e992-4ae6-8614-b4755d9986f7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:33.153735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7221ec4c-e992-4ae6-8614-b4755d9986f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:33.158174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:33.184476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7221ec4c-e992-4ae6-8614-b4755d9986f7 None None] MFA Rules not processed for user `2adcb2d5a54d4ceaa65a63c7bc03f44f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:33.186277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7221ec4c-e992-4ae6-8614-b4755d9986f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:33.290040 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 822/1648] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:33 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:33.302897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:33.411842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ebcf8-f4a8-4af3-83fb-f00fa30b2704 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:33.422090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:33.422731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:33.423035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:33.423288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:33.507975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:33.519705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:33.521161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:33.531397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 827/1649] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:35.650144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7221ec4c-e992-4ae6-8614-b4755d9986f7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:35.660869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7221ec4c-e992-4ae6-8614-b4755d9986f7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:35.665013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d479e8f-db09-4e0b-a633-2a8b5b798aac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:35.665673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d479e8f-db09-4e0b-a633-2a8b5b798aac service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:35.665949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d479e8f-db09-4e0b-a633-2a8b5b798aac service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:35.666229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d479e8f-db09-4e0b-a633-2a8b5b798aac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:35.719408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:35.735202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d479e8f-db09-4e0b-a633-2a8b5b798aac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:35.736717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d479e8f-db09-4e0b-a633-2a8b5b798aac service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:35.746629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 823/1650] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:35 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:36.816560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:36.818409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:36.953228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2a6042-3d77-4c49-b057-b7e80fdc2a3e service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:36.958934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:36.959699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:36.959990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:36.960266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] PATH_INFO: `/v3/users/797382c3e1dc454faed3eba66ec1196c` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:36.961475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=797382c3e1dc454faed3eba66ec1196c)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:36.963442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:37.172338 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '797382c3e1dc454faed3eba66ec1196c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:37.173253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '797382c3e1dc454faed3eba66ec1196c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:37.191405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '797382c3e1dc454faed3eba66ec1196c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:37.197036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:37.202024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '797382c3e1dc454faed3eba66ec1196c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:37.203351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '797382c3e1dc454faed3eba66ec1196c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:37.319410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 828/1651] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:36 2022] DELETE /identity/v3/users/797382c3e1dc454faed3eba66ec1196c => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:37.321456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:37.322369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:37.322683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:37.346327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:37.347753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:37.455521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 824/1652] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:37.473959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:37.476252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:37.505041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:37.506590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:37.550065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4518a2-344a-4eb4-ab1b-2916fe1374ef admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:37.553748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:37.554362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:37.554638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:37.554909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:37.566436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:37.567880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:37.578311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 829/1653] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:37.584896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f1ec4b-fa14-40eb-9934-b860b05a62a3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:37.592780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:37.593517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:37.593806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:37.594083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:37.595910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:37.674594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:37.677463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:37.691599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 825/1654] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:37 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:38.082523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:38.082523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:38.100237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e6392-0a4e-424a-a9a6-d37e76fd91a5 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:38.112340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:38.112995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:38.113285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:38.113599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] PATH_INFO: `/v3/projects/62895f66a6414c40b35e83680c1809eb` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:38.114837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] RBAC: Authorizing `identity:delete_project(project_id=62895f66a6414c40b35e83680c1809eb)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:38.116961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:38.459801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62895f66a6414c40b35e83680c1809eb', 'request_id': 'req-1c178344-9f4a-4237-9fdb-a84ad92cd64e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.465496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62895f66a6414c40b35e83680c1809eb', 'request_id': 'req-1c178344-9f4a-4237-9fdb-a84ad92cd64e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.473357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62895f66a6414c40b35e83680c1809eb', 'request_id': 'req-1c178344-9f4a-4237-9fdb-a84ad92cd64e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.595760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] The token cache is being invalidate because project 62895f66a6414c40b35e83680c1809eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:38.596185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.596831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.598374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 830/1655] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:38 2022] DELETE /identity/v3/projects/62895f66a6414c40b35e83680c1809eb => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:38.602347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:38.615452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7342a3e-0d46-449f-b0cc-00b0889610fe service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:38.662578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:38.688239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:38.688876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:38.689157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:38.689460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] PATH_INFO: `/v3/users/d1f97989d05b412e94009df6f3a0c485` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:38.690564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1f97989d05b412e94009df6f3a0c485)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:38.692564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:38.874393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1f97989d05b412e94009df6f3a0c485'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.886486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1f97989d05b412e94009df6f3a0c485'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.967537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1f97989d05b412e94009df6f3a0c485'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.968160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1f97989d05b412e94009df6f3a0c485'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.968788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1f97989d05b412e94009df6f3a0c485'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:38.985649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 826/1656] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:38 2022] DELETE /identity/v3/users/d1f97989d05b412e94009df6f3a0c485 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:39.415040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:39.417609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:39.437565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c178344-9f4a-4237-9fdb-a84ad92cd64e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:39.508573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:39.509190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:39.509508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:39.514200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] PATH_INFO: `/v3/projects/539bf4ccc56446da9be4008e472e307a` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:39.514200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] RBAC: Authorizing `identity:delete_project(project_id=539bf4ccc56446da9be4008e472e307a)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:39.514200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:39.530397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:39.650909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '539bf4ccc56446da9be4008e472e307a', 'request_id': 'req-b599fa08-ac14-442f-831b-64bfc799d583'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:39.659501 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '539bf4ccc56446da9be4008e472e307a', 'request_id': 'req-b599fa08-ac14-442f-831b-64bfc799d583'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:39.666896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '539bf4ccc56446da9be4008e472e307a', 'request_id': 'req-b599fa08-ac14-442f-831b-64bfc799d583'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:39.752963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] The token cache is being invalidate because project 539bf4ccc56446da9be4008e472e307a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:39.753365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:39.754138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:39.755948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 831/1657] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:39 2022] DELETE /identity/v3/projects/539bf4ccc56446da9be4008e472e307a => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:39.759856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:39.761525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:39.777264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4193df9b-8927-45bc-8fc0-c7aed656f10e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:39.856134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:39.857666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:39.859048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:39.859453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] PATH_INFO: `/v3/users/494ecbada68c4178bfd050bb42e3dce4` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:39.860957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] RBAC: Authorizing `identity:delete_user(user_id=494ecbada68c4178bfd050bb42e3dce4)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:39.863635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:40.118762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '494ecbada68c4178bfd050bb42e3dce4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:40.133589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=18,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:40.143841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '494ecbada68c4178bfd050bb42e3dce4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:40.236637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '494ecbada68c4178bfd050bb42e3dce4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:40.237293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '494ecbada68c4178bfd050bb42e3dce4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:40.237934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec764a17-9d79-4deb-a1b1-b738d8eafc60 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '494ecbada68c4178bfd050bb42e3dce4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:40.252385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 827/1658] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:39 2022] DELETE /identity/v3/users/494ecbada68c4178bfd050bb42e3dce4 => generated 0 bytes in 493 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:40.728505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:40.730014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:40.743914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b599fa08-ac14-442f-831b-64bfc799d583 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:40.805508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:40.806237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:40.806518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:40.806789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] PATH_INFO: `/v3/projects/eea0d076a3ef493baa08f9cd77410e4c` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:40.807910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eea0d076a3ef493baa08f9cd77410e4c)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:40.810423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:40.932235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eea0d076a3ef493baa08f9cd77410e4c', 'request_id': 'req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:40.938691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eea0d076a3ef493baa08f9cd77410e4c', 'request_id': 'req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:40.945153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eea0d076a3ef493baa08f9cd77410e4c', 'request_id': 'req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:41.018660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] The token cache is being invalidate because project eea0d076a3ef493baa08f9cd77410e4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:41.023880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:41.023880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fc7eabb1-9d0e-449d-9fd8-1a202dd11388 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:41.023880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 832/1659] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:40 2022] DELETE /identity/v3/projects/eea0d076a3ef493baa08f9cd77410e4c => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:41.031256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f449b581-a206-405b-8e98-ca3f6570758c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.031763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f449b581-a206-405b-8e98-ca3f6570758c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.031863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f449b581-a206-405b-8e98-ca3f6570758c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.054389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f449b581-a206-405b-8e98-ca3f6570758c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:41.055796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f449b581-a206-405b-8e98-ca3f6570758c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:41.137249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 828/1660] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:41.139394 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055e9576-024d-4e38-ba6d-0f4a978c0ff2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.139672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055e9576-024d-4e38-ba6d-0f4a978c0ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.140037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055e9576-024d-4e38-ba6d-0f4a978c0ff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.162365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-055e9576-024d-4e38-ba6d-0f4a978c0ff2 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:41.163622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-055e9576-024d-4e38-ba6d-0f4a978c0ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:41.173066 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 833/1661] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:41.177271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f449b581-a206-405b-8e98-ca3f6570758c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:41.188772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f449b581-a206-405b-8e98-ca3f6570758c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:41.193948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:41.195485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.195485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.195485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.196153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:41.196555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:41.198457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:41.217545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 829/1662] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:06:41.230369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055e9576-024d-4e38-ba6d-0f4a978c0ff2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:41.240334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055e9576-024d-4e38-ba6d-0f4a978c0ff2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:41.245311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:41.246028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.246217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.246489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.247149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:41.249173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:41.357960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 834/1663] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:41 2022] POST /identity/v3/projects => generated 383 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:41.363324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:41.374071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d977c4e-c476-44fa-ada1-44e3135e6891 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:41.452267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:41.452888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.453173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.453491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.454161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:41.456082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:41.565566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 830/1664] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:41 2022] POST /identity/v3/users => generated 292 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:41.567052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:41.574210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:41.578589 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7c562b-3445-4422-ae91-0777561ef5ec admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:41.583092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:41.586716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.586716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.586716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.586716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:41.586716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:41.597246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 835/1665] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:41.601604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:41.612730 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c2538e-2d91-4523-b06b-39097f75f799 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:41.617689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098bb109-cbf5-465c-92e0-4917eeb59695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:41.618379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098bb109-cbf5-465c-92e0-4917eeb59695 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.618726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098bb109-cbf5-465c-92e0-4917eeb59695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.619063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098bb109-cbf5-465c-92e0-4917eeb59695 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.619768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098bb109-cbf5-465c-92e0-4917eeb59695 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:41.621459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098bb109-cbf5-465c-92e0-4917eeb59695 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:41.634348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 831/1666] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:41.641844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:41.652882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff5e1f0-8362-447a-9123-4bba6d0fab52 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:41.658128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:41.659104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.659386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.659691 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] PATH_INFO: `/v3/projects/402072ccf4f9459c835562dbd9e5fdd2/users/d5e43659aea943eabd5404406131b926/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.675564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] RBAC: Authorizing `identity:create_grant(project_id=402072ccf4f9459c835562dbd9e5fdd2, user_id=d5e43659aea943eabd5404406131b926, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:41.678316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:41.785016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 836/1667] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:41 2022] PUT /identity/v3/projects/402072ccf4f9459c835562dbd9e5fdd2/users/d5e43659aea943eabd5404406131b926/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:41.798798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.799094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.799380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.829904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] MFA Rules not processed for user `d5e43659aea943eabd5404406131b926`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:41.831187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:41.864985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:41.901771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 832/1668] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:41.914482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:41.916189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:41.977212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e169a41-327e-417a-8f12-5824d8ccc956 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:41.981041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:41.982165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:41.982460 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:41.982743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:41.991272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:41.993706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:42.002447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 837/1669] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:41 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:44.584045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:44.585621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:44.675888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387976dd-0835-41a3-85eb-3e6ad7b17e3b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:44.680270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:44.680921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:44.681201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:44.681518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] PATH_INFO: `/v3/users/fce4e45562744a6d98acd0addfdf1b18` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:44.682654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] RBAC: Authorizing `identity:delete_user(user_id=fce4e45562744a6d98acd0addfdf1b18)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:44.689341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:44.716898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:44.719035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:44.734463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e3ccdb-530f-4382-b716-f3fcb3a88fd9 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:44.739379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:44.740008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:44.740321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:44.740593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] PATH_INFO: `/v3/users/2c8d0df59a54449485d86b8533496d1f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:44.741724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c8d0df59a54449485d86b8533496d1f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:44.743633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:44.907821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fce4e45562744a6d98acd0addfdf1b18'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:44.920249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fce4e45562744a6d98acd0addfdf1b18'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:44.992372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c8d0df59a54449485d86b8533496d1f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:44.993036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c8d0df59a54449485d86b8533496d1f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.001908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fce4e45562744a6d98acd0addfdf1b18'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.002595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fce4e45562744a6d98acd0addfdf1b18'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.003318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72461a7b-ed79-44cf-aa1c-ad5c9a85d44e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fce4e45562744a6d98acd0addfdf1b18'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.018058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c8d0df59a54449485d86b8533496d1f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.024606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 833/1670] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:44 2022] DELETE /identity/v3/users/fce4e45562744a6d98acd0addfdf1b18 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:45.027018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c8d0df59a54449485d86b8533496d1f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.029423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7e95b93c-70d5-4717-95b5-40ca900f596d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c8d0df59a54449485d86b8533496d1f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.029560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a13678-ed81-4333-911e-671831687bfa None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:45.033727 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a13678-ed81-4333-911e-671831687bfa None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:45.033727 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a13678-ed81-4333-911e-671831687bfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:45.055814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-03a13678-ed81-4333-911e-671831687bfa None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:45.057195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-03a13678-ed81-4333-911e-671831687bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:45.096149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 838/1671] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:44 2022] DELETE /identity/v3/users/2c8d0df59a54449485d86b8533496d1f => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:45.110463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30027fb-3494-48e0-af05-44884733a52c None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:45.111123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30027fb-3494-48e0-af05-44884733a52c None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:45.111606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30027fb-3494-48e0-af05-44884733a52c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:45.123882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:45.146885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a30027fb-3494-48e0-af05-44884733a52c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:45.147086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 834/1672] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:45.147226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a30027fb-3494-48e0-af05-44884733a52c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:45.163328 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a13678-ed81-4333-911e-671831687bfa None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:45.164798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-03a13678-ed81-4333-911e-671831687bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:45.219487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:45.240152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 839/1673] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:45.241202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30027fb-3494-48e0-af05-44884733a52c None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:45.242575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a30027fb-3494-48e0-af05-44884733a52c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:45.256941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30027fb-3494-48e0-af05-44884733a52c None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:45.262025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:45.262681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:45.263072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:45.263503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] PATH_INFO: `/v3/users/d5e43659aea943eabd5404406131b926` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:45.264385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a13678-ed81-4333-911e-671831687bfa None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:45.265724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d5e43659aea943eabd5404406131b926)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:45.269051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:45.335240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:45.339032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:45.339032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:45.339032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:45.348457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:45.350569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:45.360986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 835/1674] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:45.361686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:45.370726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee38ce-2b3d-4b93-99af-28a973fd7fb0 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:45.375631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3694690-18ac-40ed-b4d1-29d57539bf0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:45.376249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3694690-18ac-40ed-b4d1-29d57539bf0a service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:45.376541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3694690-18ac-40ed-b4d1-29d57539bf0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:45.377383 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3694690-18ac-40ed-b4d1-29d57539bf0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:45.388972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3694690-18ac-40ed-b4d1-29d57539bf0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:45.390531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3694690-18ac-40ed-b4d1-29d57539bf0a service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:45.409000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 836/1675] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:45.458392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd5e43659aea943eabd5404406131b926'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.459170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd5e43659aea943eabd5404406131b926'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.477504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd5e43659aea943eabd5404406131b926'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.491146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd5e43659aea943eabd5404406131b926'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.491987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd5e43659aea943eabd5404406131b926'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:45.618275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 840/1676] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:45 2022] DELETE /identity/v3/users/d5e43659aea943eabd5404406131b926 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:45.623281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:45.623845 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:45.624211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:45.659209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:45.661577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:45.746599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 837/1677] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:45.756036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:45.757597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:45.837240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b734b7-ecc9-4864-ba23-677b45507626 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:45.841905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:45.842550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:45.842837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:45.843110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:45.867281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:45.879360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:45.885773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 841/1678] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:46.257658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:46.259212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:46.277893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6739b53f-38c4-4053-a04d-02eb498dc71d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:46.286376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:46.287053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:46.287337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:46.287618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] PATH_INFO: `/v3/projects/fd351d224f27454e96502e67e9a89873` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:46.288824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd351d224f27454e96502e67e9a89873)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:46.290966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:46.420970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:46.446447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd351d224f27454e96502e67e9a89873', 'request_id': 'req-9403c19c-7e9e-4217-b10e-cb6c05e098fd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.522071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd351d224f27454e96502e67e9a89873', 'request_id': 'req-9403c19c-7e9e-4217-b10e-cb6c05e098fd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.530020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd351d224f27454e96502e67e9a89873', 'request_id': 'req-9403c19c-7e9e-4217-b10e-cb6c05e098fd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.556591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] The token cache is being invalidate because project fd351d224f27454e96502e67e9a89873 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:46.556591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.557560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.560654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 838/1679] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:46 2022] DELETE /identity/v3/projects/fd351d224f27454e96502e67e9a89873 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:46.565353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:46.579288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2a79dd-177d-4463-8a21-cf5ff952d867 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:46.603070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:46.603070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:46.690637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:46.697197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:46.697197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:46.697197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] PATH_INFO: `/v3/users/5f4ebc42a8364c6bb4ab5f30a01a5025` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:46.697197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f4ebc42a8364c6bb4ab5f30a01a5025)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:46.697197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:46.700145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:46.728155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403c19c-7e9e-4217-b10e-cb6c05e098fd admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:46.738196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:46.738933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:46.739213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:46.739701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] PATH_INFO: `/v3/projects/089c3548781b4dddac6796d8da77b562` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:46.741498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] RBAC: Authorizing `identity:delete_project(project_id=089c3548781b4dddac6796d8da77b562)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:46.744685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:46.909729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f4ebc42a8364c6bb4ab5f30a01a5025'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.910374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '089c3548781b4dddac6796d8da77b562', 'request_id': 'req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.910523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f4ebc42a8364c6bb4ab5f30a01a5025'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.923218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f4ebc42a8364c6bb4ab5f30a01a5025'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.929978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f4ebc42a8364c6bb4ab5f30a01a5025'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.932404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f4ebc42a8364c6bb4ab5f30a01a5025'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:46.988093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '089c3548781b4dddac6796d8da77b562', 'request_id': 'req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.005260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '089c3548781b4dddac6796d8da77b562', 'request_id': 'req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.039386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] The token cache is being invalidate because project 089c3548781b4dddac6796d8da77b562 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:47.039764 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.040629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.042270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 839/1680] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:46 2022] DELETE /identity/v3/projects/089c3548781b4dddac6796d8da77b562 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:47.043101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:47.045985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:47.055245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 842/1681] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:46 2022] DELETE /identity/v3/users/5f4ebc42a8364c6bb4ab5f30a01a5025 => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:47.055245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:47.056854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:47.155442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:47.182929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1991f6-f1f8-41c8-8bf4-ee5fcb150353 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:47.219105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006830d4-70c7-4007-a272-96f2c53e0792 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:47.227048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:47.228150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:47.228577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:47.229284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] PATH_INFO: `/v3/users/aee73d6b32194d1487abf1e309af9dcd` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:47.232244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aee73d6b32194d1487abf1e309af9dcd)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:47.235916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:47.259833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:47.260611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:47.260967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:47.261347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] PATH_INFO: `/v3/projects/402072ccf4f9459c835562dbd9e5fdd2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:47.262581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=402072ccf4f9459c835562dbd9e5fdd2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:47.265350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:47.443567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aee73d6b32194d1487abf1e309af9dcd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.444552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '402072ccf4f9459c835562dbd9e5fdd2', 'request_id': 'req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.445136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aee73d6b32194d1487abf1e309af9dcd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.467671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aee73d6b32194d1487abf1e309af9dcd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.477696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aee73d6b32194d1487abf1e309af9dcd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.478850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-60da0d9c-506e-44f7-ab66-939dbe9f5318 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aee73d6b32194d1487abf1e309af9dcd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.510317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '402072ccf4f9459c835562dbd9e5fdd2', 'request_id': 'req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.520869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '402072ccf4f9459c835562dbd9e5fdd2', 'request_id': 'req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.539622 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] The token cache is being invalidate because project 402072ccf4f9459c835562dbd9e5fdd2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:47.540043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.541088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d22edcd2-28a2-416a-ae8e-7ad289c57dc6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.542605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 840/1682] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:47 2022] DELETE /identity/v3/projects/402072ccf4f9459c835562dbd9e5fdd2 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:47.549184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:47.549509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:47.549796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:47.581817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:47.583369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:47.596536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:47.602480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 843/1683] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:47 2022] DELETE /identity/v3/users/aee73d6b32194d1487abf1e309af9dcd => generated 0 bytes in 548 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:47.660183 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 841/1684] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:47.667965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1afecf3-501c-41fb-904b-c664d5a6ed01 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:47.668249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1afecf3-501c-41fb-904b-c664d5a6ed01 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:47.668519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1afecf3-501c-41fb-904b-c664d5a6ed01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:47.686796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:47.688759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:47.692390 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e1afecf3-501c-41fb-904b-c664d5a6ed01 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:47.694005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e1afecf3-501c-41fb-904b-c664d5a6ed01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:47.713510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2256da1-10cf-4818-bcad-0ed3d7c5ba6c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:47.751621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:47.780184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 844/1685] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:47.784713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1afecf3-501c-41fb-904b-c664d5a6ed01 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:47.786001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:47.786653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:47.786936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:47.787213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] PATH_INFO: `/v3/projects/36cdb00e138e4fdd9c6cfdfd28ddcfec` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:47.788320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36cdb00e138e4fdd9c6cfdfd28ddcfec)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:47.790349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:47.793676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1afecf3-501c-41fb-904b-c664d5a6ed01 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:47.799862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:47.805695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:47.805695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:47.805695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:47.805695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:47.805695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:47.805695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:47.816543 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 845/1686] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:47.820826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:47.838135 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d3e192-6140-4019-8b80-bfd60f32fc00 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:47.843503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:47.843952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:47.844231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:47.844510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:47.845170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:47.847164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:47.940189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36cdb00e138e4fdd9c6cfdfd28ddcfec', 'request_id': 'req-3b1917a4-aaf8-489a-b363-e3901cd6c672'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:47.971169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 846/1687] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:47 2022] POST /identity/v3/projects => generated 383 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:47.975099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.042563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36cdb00e138e4fdd9c6cfdfd28ddcfec', 'request_id': 'req-3b1917a4-aaf8-489a-b363-e3901cd6c672'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.053246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36cdb00e138e4fdd9c6cfdfd28ddcfec', 'request_id': 'req-3b1917a4-aaf8-489a-b363-e3901cd6c672'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.071937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] The token cache is being invalidate because project 36cdb00e138e4fdd9c6cfdfd28ddcfec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:48.072248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.072638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6fbebb-79a2-4881-b793-d91d4ce21272 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.073657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3b1917a4-aaf8-489a-b363-e3901cd6c672 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.080912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 842/1688] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:47 2022] DELETE /identity/v3/projects/36cdb00e138e4fdd9c6cfdfd28ddcfec => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:48.082780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5d7185-df2a-44e0-8980-b1c8dcb8b518 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.083073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5d7185-df2a-44e0-8980-b1c8dcb8b518 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.083585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5d7185-df2a-44e0-8980-b1c8dcb8b518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.084061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.084939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.085507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.086012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.086856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.089203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.108603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3b5d7185-df2a-44e0-8980-b1c8dcb8b518 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:48.109879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b5d7185-df2a-44e0-8980-b1c8dcb8b518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.129400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 843/1689] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:48.135341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9124fc-f79a-49f2-aae6-598a871c1a65 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.135635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9124fc-f79a-49f2-aae6-598a871c1a65 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.135916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9124fc-f79a-49f2-aae6-598a871c1a65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.159862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4c9124fc-f79a-49f2-aae6-598a871c1a65 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:48.161076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4c9124fc-f79a-49f2-aae6-598a871c1a65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.169777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 844/1690] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:48.173651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9124fc-f79a-49f2-aae6-598a871c1a65 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.185514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9124fc-f79a-49f2-aae6-598a871c1a65 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.191268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.191937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.201109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.201109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.201109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:48.201109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.201455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 847/1691] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:47 2022] POST /identity/v3/users => generated 292 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:48.201545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.204410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.206361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.221348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 845/1692] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:48.225174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.225799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9632dc90-dc73-4724-abdb-b28fc94d04ae admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.238019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd935a63-fa41-4946-9aca-14f942b2ee98 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.241867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.242905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.243340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.244101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.244411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.244775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.245307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.245442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.245682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.246321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.248161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.249270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.270045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 848/1693] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:48.270045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.279662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20554e1-9b62-4f7d-a959-d35675cef2cf admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.284893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.285363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.285841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.285968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.286554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.288475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.300432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 849/1694] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:48.304693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.315531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af99b4a1-51b7-4f03-ae12-2fcf97bcdedf admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.320348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.320993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.321281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.321573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] PATH_INFO: `/v3/projects/62f0f20404ba41d3aa380a9d10eeab81/users/14729da7649a4ab785ff87981968d342/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.337160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] RBAC: Authorizing `identity:create_grant(project_id=62f0f20404ba41d3aa380a9d10eeab81, user_id=14729da7649a4ab785ff87981968d342, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.339861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.358755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 846/1695] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:48 2022] POST /identity/v3/projects => generated 367 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:48.359624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.361414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.428054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b291b-0e37-4c79-811d-7df9df3bb067 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.433371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.435279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.435754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.438026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] PATH_INFO: `/v3/projects/be99d0e1d4904f8885593c32f2e0b839` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.441250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be99d0e1d4904f8885593c32f2e0b839)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.442066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.451215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 850/1696] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:48 2022] PUT /identity/v3/projects/62f0f20404ba41d3aa380a9d10eeab81/users/14729da7649a4ab785ff87981968d342/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:48.451215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.455468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:48.465132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:48.525602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86acd68-8596-4dcf-9fcd-8f11c00ba95b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.530377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62758b3-a851-4742-a6a0-542bec7cc334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.530985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62758b3-a851-4742-a6a0-542bec7cc334 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.531271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62758b3-a851-4742-a6a0-542bec7cc334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.531545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62758b3-a851-4742-a6a0-542bec7cc334 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.532172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62758b3-a851-4742-a6a0-542bec7cc334 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.534306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62758b3-a851-4742-a6a0-542bec7cc334 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.585116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be99d0e1d4904f8885593c32f2e0b839', 'request_id': 'req-5211cfd2-531b-4ced-b330-309eeaaf7704'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.675548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 851/1697] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:48 2022] POST /identity/v3/users => generated 284 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:48.677843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.678137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.678422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.689079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be99d0e1d4904f8885593c32f2e0b839', 'request_id': 'req-5211cfd2-531b-4ced-b330-309eeaaf7704'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.700745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be99d0e1d4904f8885593c32f2e0b839', 'request_id': 'req-5211cfd2-531b-4ced-b330-309eeaaf7704'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.704438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:48.705276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.723130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] The token cache is being invalidate because project be99d0e1d4904f8885593c32f2e0b839 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:48.723804 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.725127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:48.727710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 847/1698] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:48 2022] DELETE /identity/v3/projects/be99d0e1d4904f8885593c32f2e0b839 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:48.729184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.732724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.801718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 852/1699] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:48.802971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.804815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.818818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4add1714-ee09-46ac-b1f4-346b00b08f7e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.823773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.824398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.824675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.824945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] PATH_INFO: `/v3/users/2adcb2d5a54d4ceaa65a63c7bc03f44f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.826293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2adcb2d5a54d4ceaa65a63c7bc03f44f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.826401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211cfd2-531b-4ced-b330-309eeaaf7704 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.828247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.832683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88920765-bd43-4df9-9cc4-29981ab12608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.833131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88920765-bd43-4df9-9cc4-29981ab12608 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.833590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88920765-bd43-4df9-9cc4-29981ab12608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.833904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88920765-bd43-4df9-9cc4-29981ab12608 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.834638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88920765-bd43-4df9-9cc4-29981ab12608 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.836500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88920765-bd43-4df9-9cc4-29981ab12608 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.847463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 848/1700] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:48.848694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e96de4-8757-4624-8401-5f49ba192fa8 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.849110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e96de4-8757-4624-8401-5f49ba192fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.849449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e96de4-8757-4624-8401-5f49ba192fa8 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.850733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 849/1701] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:06:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:06:48.851459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e96de4-8757-4624-8401-5f49ba192fa8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.853437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-50e96de4-8757-4624-8401-5f49ba192fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:48.915565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e96de4-8757-4624-8401-5f49ba192fa8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.919104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.920698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.920698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.920698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.931597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.935847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.951350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 850/1702] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:48.951350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:48.959904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c94498-c812-4dfe-97ae-d16ba2bc1365 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:48.965237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89722dd8-7392-4e56-b390-6f0eda1758a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:48.967555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89722dd8-7392-4e56-b390-6f0eda1758a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.968059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89722dd8-7392-4e56-b390-6f0eda1758a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.968996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89722dd8-7392-4e56-b390-6f0eda1758a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:48.969889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89722dd8-7392-4e56-b390-6f0eda1758a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:48.972984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89722dd8-7392-4e56-b390-6f0eda1758a1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:48.987123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 851/1703] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:48 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:48.989710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:48.989710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:48.996612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.013487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:49.014833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.050136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2adcb2d5a54d4ceaa65a63c7bc03f44f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.050834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2adcb2d5a54d4ceaa65a63c7bc03f44f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.063132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2adcb2d5a54d4ceaa65a63c7bc03f44f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.070172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2adcb2d5a54d4ceaa65a63c7bc03f44f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.070902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2adcb2d5a54d4ceaa65a63c7bc03f44f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.081255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 852/1704] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:06:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:49.082143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:49.083762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.097401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd51d4a-45c7-4395-90e0-b27940021e7f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:49.102094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:49.102725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.103014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.103283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] PATH_INFO: `/v3/users/1b3396bc67ae4f6c9c8808653fef4676` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.104392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b3396bc67ae4f6c9c8808653fef4676)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:49.106352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:49.150101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 853/1705] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:48 2022] DELETE /identity/v3/users/2adcb2d5a54d4ceaa65a63c7bc03f44f => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:49.151137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:49.152621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.164296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:49.228723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbd0581-1770-45e2-9471-508d76dfe0f9 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:49.233976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea2a1f-daf2-40d7-a1b1-e148b75be73f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:49.234642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea2a1f-daf2-40d7-a1b1-e148b75be73f admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.234919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea2a1f-daf2-40d7-a1b1-e148b75be73f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.235192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea2a1f-daf2-40d7-a1b1-e148b75be73f admin admin] PATH_INFO: `/v3/projects/bbc0a39dd74649989d17306408e4b90c/users/34940012678441ac88b898dbd0d323db/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.250474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea2a1f-daf2-40d7-a1b1-e148b75be73f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bbc0a39dd74649989d17306408e4b90c, user_id=34940012678441ac88b898dbd0d323db, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:49.253138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea2a1f-daf2-40d7-a1b1-e148b75be73f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:49.267502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b3396bc67ae4f6c9c8808653fef4676'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.280154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b3396bc67ae4f6c9c8808653fef4676'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.323802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:49.326357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b3396bc67ae4f6c9c8808653fef4676'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.327952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 854/1706] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:49 2022] PUT /identity/v3/projects/bbc0a39dd74649989d17306408e4b90c/users/34940012678441ac88b898dbd0d323db/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:49.329111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b3396bc67ae4f6c9c8808653fef4676'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.329211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd30898-3664-4f20-8fc5-23b4f9db8d92 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.329286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec9ad964-3bca-47e3-98c9-5e570cffb879 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b3396bc67ae4f6c9c8808653fef4676'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.329530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd30898-3664-4f20-8fc5-23b4f9db8d92 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.329530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd30898-3664-4f20-8fc5-23b4f9db8d92 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.330880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 855/1707] 10.222.0.66 () {54 vars in 961 bytes} [Fri Oct 28 20:06:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:06:49.339889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.340175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.340451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.348058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 853/1708] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:49 2022] DELETE /identity/v3/users/1b3396bc67ae4f6c9c8808653fef4676 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:49.350622 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c1ef9f-6e8c-479c-9e7d-f9474f002f57 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.351180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c1ef9f-6e8c-479c-9e7d-f9474f002f57 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.351529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c1ef9f-6e8c-479c-9e7d-f9474f002f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.374459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-48c1ef9f-6e8c-479c-9e7d-f9474f002f57 None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:49.375945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-48c1ef9f-6e8c-479c-9e7d-f9474f002f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.387287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] MFA Rules not processed for user `34940012678441ac88b898dbd0d323db`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:49.389503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.482143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 856/1709] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:49 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:49.487677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:49.490218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.490829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 854/1710] 10.222.0.66 () {58 vars in 1058 bytes} [Fri Oct 28 20:06:49 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:49.501609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c1ef9f-6e8c-479c-9e7d-f9474f002f57 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:49.512714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c1ef9f-6e8c-479c-9e7d-f9474f002f57 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:49.519581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:49.523107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.523107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.523107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.525344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.541185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:49.542797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:49.557181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 855/1711] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:49.579685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306a5f52-5ae1-40d8-807b-a406bf8e9420 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:49.585951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:49.586147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.586414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.586701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.587548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:49.589485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:49.623615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:49.626135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.645862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c416703-3a68-4f82-aa1c-64139115c981 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:49.655167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:49.655958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.656251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.656579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] PATH_INFO: `/v3/projects/e00d29c292fe43b7abc29883ca18cdbe` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.657969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e00d29c292fe43b7abc29883ca18cdbe)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:49.660211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:49.693092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 857/1712] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:49 2022] POST /identity/v3/projects => generated 367 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:49.696818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:49.775742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6b3ec7-9cfb-4b9a-b3a5-a6e7ec7b8cbe admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:49.777547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e00d29c292fe43b7abc29883ca18cdbe', 'request_id': 'req-ab8df9aa-610f-46ac-a590-f1c300124392'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.846022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80917e48-1aaa-4b2f-a3e7-6cce942e0574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:49.846646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80917e48-1aaa-4b2f-a3e7-6cce942e0574 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.846933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80917e48-1aaa-4b2f-a3e7-6cce942e0574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.847209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80917e48-1aaa-4b2f-a3e7-6cce942e0574 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.847949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80917e48-1aaa-4b2f-a3e7-6cce942e0574 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:49.849915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80917e48-1aaa-4b2f-a3e7-6cce942e0574 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:49.875851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e00d29c292fe43b7abc29883ca18cdbe', 'request_id': 'req-ab8df9aa-610f-46ac-a590-f1c300124392'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.883417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e00d29c292fe43b7abc29883ca18cdbe', 'request_id': 'req-ab8df9aa-610f-46ac-a590-f1c300124392'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.896142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] The token cache is being invalidate because project e00d29c292fe43b7abc29883ca18cdbe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:49.896556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.897532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:49.899651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 856/1713] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:49 2022] DELETE /identity/v3/projects/e00d29c292fe43b7abc29883ca18cdbe => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:49.900815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:49.902764 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:49.918379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8df9aa-610f-46ac-a590-f1c300124392 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:49.924859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:49.925621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:49.925917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:49.926200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] PATH_INFO: `/v3/projects/40fa4f575da04bc39b8df1e74fb77035` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:49.928300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40fa4f575da04bc39b8df1e74fb77035)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:49.930435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.020830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 858/1714] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:49 2022] POST /identity/v3/users => generated 290 bytes in 322 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:50.020830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.020830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.020830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.046005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:50.047287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.052502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40fa4f575da04bc39b8df1e74fb77035', 'request_id': 'req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:50.113392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 859/1715] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:50.114170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.120476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.121890 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40fa4f575da04bc39b8df1e74fb77035', 'request_id': 'req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:50.131259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40fa4f575da04bc39b8df1e74fb77035', 'request_id': 'req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:50.135334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab2e012-176b-487c-9415-168a0ad91f43 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.139733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432d7fe3-ede2-4859-854d-1d31e5d16a35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.140437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d7fe3-ede2-4859-854d-1d31e5d16a35 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.140718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d7fe3-ede2-4859-854d-1d31e5d16a35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.140990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d7fe3-ede2-4859-854d-1d31e5d16a35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.141665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432d7fe3-ede2-4859-854d-1d31e5d16a35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.143328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432d7fe3-ede2-4859-854d-1d31e5d16a35 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.145840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] The token cache is being invalidate because project 40fa4f575da04bc39b8df1e74fb77035 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:50.146270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:50.147041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eeb4b1a1-8447-443c-ad77-7beed2fc48d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:50.149142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 857/1716] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:49 2022] DELETE /identity/v3/projects/40fa4f575da04bc39b8df1e74fb77035 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:50.152405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.152405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.152405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.161200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 860/1717] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:50.161200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39edb2bb-f329-49c0-b453-e981c681caa6 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.161200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39edb2bb-f329-49c0-b453-e981c681caa6 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.161200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39edb2bb-f329-49c0-b453-e981c681caa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.173154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:50.174421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.180278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-39edb2bb-f329-49c0-b453-e981c681caa6 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:50.185426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-39edb2bb-f329-49c0-b453-e981c681caa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.185533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 858/1718] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:50.187101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.188433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.190321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 861/1719] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:50.191104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39edb2bb-f329-49c0-b453-e981c681caa6 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.203705 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be3d59-65ea-43c7-876f-b1c19d12ad63 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.204966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39edb2bb-f329-49c0-b453-e981c681caa6 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.208626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.209304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.209627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.209903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] PATH_INFO: `/v3/projects/63c2b251ad8e48c89e025c59c7f61cca/users/a569ec01c5a44f84857af4bff8ffb94c/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.211549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1727ed74-97b0-4fe0-aad8-99c38e058c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.212517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1727ed74-97b0-4fe0-aad8-99c38e058c9b admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.212784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1727ed74-97b0-4fe0-aad8-99c38e058c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.214074 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1727ed74-97b0-4fe0-aad8-99c38e058c9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.214781 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1727ed74-97b0-4fe0-aad8-99c38e058c9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:50.215090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1727ed74-97b0-4fe0-aad8-99c38e058c9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.219468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1727ed74-97b0-4fe0-aad8-99c38e058c9b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.225654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63c2b251ad8e48c89e025c59c7f61cca, user_id=a569ec01c5a44f84857af4bff8ffb94c, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.228298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.229073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 862/1720] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:50.230536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaa42f-e07b-4ec6-9848-fc1ed25b45d3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.230749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaa42f-e07b-4ec6-9848-fc1ed25b45d3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.230973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaa42f-e07b-4ec6-9848-fc1ed25b45d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.250910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5eeaa42f-e07b-4ec6-9848-fc1ed25b45d3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:50.252113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5eeaa42f-e07b-4ec6-9848-fc1ed25b45d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.267157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:50.268976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 863/1721] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:50.269861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeaa42f-e07b-4ec6-9848-fc1ed25b45d3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.279774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeaa42f-e07b-4ec6-9848-fc1ed25b45d3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.284773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.285402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.285791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.291340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.291340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.291530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 859/1722] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:50 2022] PUT /identity/v3/projects/63c2b251ad8e48c89e025c59c7f61cca/users/a569ec01c5a44f84857af4bff8ffb94c/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:50.291623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.291708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.360635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ef707e-77fa-4c5b-aa55-d82d67765f70 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.364987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 864/1723] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:50.372806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4476c5b2-7ee8-401a-bae3-60b96bd33118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.372806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476c5b2-7ee8-401a-bae3-60b96bd33118 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.373031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.373123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476c5b2-7ee8-401a-bae3-60b96bd33118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.373123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476c5b2-7ee8-401a-bae3-60b96bd33118 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.373123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476c5b2-7ee8-401a-bae3-60b96bd33118 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:50.373123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476c5b2-7ee8-401a-bae3-60b96bd33118 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.373648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:50.374507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476c5b2-7ee8-401a-bae3-60b96bd33118 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.389586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 860/1724] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:50.391200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b456da-5f72-4050-9bad-c261b08748c7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.391573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b456da-5f72-4050-9bad-c261b08748c7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.391906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b456da-5f72-4050-9bad-c261b08748c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.422614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-12b456da-5f72-4050-9bad-c261b08748c7 None None] MFA Rules not processed for user `14729da7649a4ab785ff87981968d342`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:50.423950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-12b456da-5f72-4050-9bad-c261b08748c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.444435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f60078-20c3-4655-bc4f-15354c32b1e6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.450484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.451789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.452253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.452678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.453566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.456284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.469055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 865/1725] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:50.469900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.482447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df0c7fc-e69f-4bc1-a0b0-771780124fb0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.487201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.487804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.488132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.488413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.489075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.491034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.500493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 861/1726] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:50.501683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b456da-5f72-4050-9bad-c261b08748c7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.513762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b456da-5f72-4050-9bad-c261b08748c7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.518483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.519089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.522479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.522479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.522479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.522479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.645569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 866/1727] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/users => generated 293 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:50.645569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.653024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804d5ee-d16f-44a1-9b99-c0027112c059 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.658326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.658969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.659253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.659532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] PATH_INFO: `/v3/projects/63c2b251ad8e48c89e025c59c7f61cca/users/a569ec01c5a44f84857af4bff8ffb94c/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.661470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63c2b251ad8e48c89e025c59c7f61cca, user_id=a569ec01c5a44f84857af4bff8ffb94c, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.664180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.676038 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 867/1728] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:50 2022] PUT /identity/v3/projects/63c2b251ad8e48c89e025c59c7f61cca/users/a569ec01c5a44f84857af4bff8ffb94c/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:50.685411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.685411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.695141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 862/1729] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/projects => generated 373 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:50.698830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.739553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eaa35d-b5b1-4626-a0cb-537ecd690c13 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.760347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42704925-b59e-461b-abfb-6fdda0b08157 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.765180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8d43f9-693c-40ee-9cc9-226579074b1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.765817 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8d43f9-693c-40ee-9cc9-226579074b1a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.766097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8d43f9-693c-40ee-9cc9-226579074b1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.766372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8d43f9-693c-40ee-9cc9-226579074b1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.767013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8d43f9-693c-40ee-9cc9-226579074b1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.768604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8d43f9-693c-40ee-9cc9-226579074b1a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.779359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 863/1730] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:50.780664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556cd17e-92ba-4669-bb45-d54d67209373 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.780959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556cd17e-92ba-4669-bb45-d54d67209373 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.783269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556cd17e-92ba-4669-bb45-d54d67209373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.793223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.793887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.794979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.795142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.799000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:50.810224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-556cd17e-92ba-4669-bb45-d54d67209373 None None] MFA Rules not processed for user `a569ec01c5a44f84857af4bff8ffb94c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:50.811431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-556cd17e-92ba-4669-bb45-d54d67209373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.857847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.859504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.868720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 868/1731] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:50.869659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.880824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c7fef2-5701-4de8-a027-d69efdc9d968 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.882782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 864/1732] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:50.883759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556cd17e-92ba-4669-bb45-d54d67209373 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.892733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.892733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.892733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.892733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.893034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.894714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.895346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556cd17e-92ba-4669-bb45-d54d67209373 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.900569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.901124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.901644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.901926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.902565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.905015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.915711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 865/1733] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:50.916225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.917923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:50.930748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496b8dae-ed76-4013-a10a-e9e808a88df4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.934388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.935605 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.935887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.936159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.939390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:50.947788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.949257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:50.959702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 866/1734] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:50 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:50.960546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:50.970533 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6276206-fa26-458a-9402-714ee3aa0c07 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:50.975389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4ebf6-be14-4884-9368-cd350d7f5757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:50.976089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4ebf6-be14-4884-9368-cd350d7f5757 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:50.976372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4ebf6-be14-4884-9368-cd350d7f5757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:50.976677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4ebf6-be14-4884-9368-cd350d7f5757 admin admin] PATH_INFO: `/v3/projects/e0b7ca901b75407ea76ec658ca4f5c70/users/3a71a3f992314746a56dee324c1f111a/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:50.994268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4ebf6-be14-4884-9368-cd350d7f5757 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0b7ca901b75407ea76ec658ca4f5c70, user_id=3a71a3f992314746a56dee324c1f111a, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:50.998480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4ebf6-be14-4884-9368-cd350d7f5757 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.020218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 869/1735] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:50 2022] POST /identity/v3/users => generated 287 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:51.024300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.035791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c6bfeb-1ba4-41e2-8a48-1ebd3d98f8c6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.040539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.041137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.041462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.041885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.042420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.044671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.055029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 870/1736] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:51.059015 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.069447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e261e67-a68f-43b6-96db-59e5a8b33a50 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.074153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.074780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.075068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.075524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.077028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.078730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.096987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 871/1737] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:51.096987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.107853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3644a4c4-6c89-4daf-a554-3318bf681410 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.113005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24db8e44-36b7-4668-98e7-7fc5d9b0fc76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.113731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24db8e44-36b7-4668-98e7-7fc5d9b0fc76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.114289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24db8e44-36b7-4668-98e7-7fc5d9b0fc76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.114377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24db8e44-36b7-4668-98e7-7fc5d9b0fc76 admin admin] PATH_INFO: `/v3/projects/801236b29780486981d6f3105f61ea56/users/aa8812ac328e45a8aae8f57d2a69efa3/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.120697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:51.127281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 867/1738] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:50 2022] PUT /identity/v3/projects/e0b7ca901b75407ea76ec658ca4f5c70/users/3a71a3f992314746a56dee324c1f111a/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:51.137163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196cffeb-b85b-4744-9360-e51efdd6b797 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.137583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196cffeb-b85b-4744-9360-e51efdd6b797 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.138102 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196cffeb-b85b-4744-9360-e51efdd6b797 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.138608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24db8e44-36b7-4668-98e7-7fc5d9b0fc76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=801236b29780486981d6f3105f61ea56, user_id=aa8812ac328e45a8aae8f57d2a69efa3, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.142348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24db8e44-36b7-4668-98e7-7fc5d9b0fc76 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.172201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-196cffeb-b85b-4744-9360-e51efdd6b797 None None] MFA Rules not processed for user `3a71a3f992314746a56dee324c1f111a`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:51.173418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-196cffeb-b85b-4744-9360-e51efdd6b797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:51.243161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 872/1739] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:51 2022] PUT /identity/v3/projects/801236b29780486981d6f3105f61ea56/users/aa8812ac328e45a8aae8f57d2a69efa3/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:51.244825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.245119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.245425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.271532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 868/1740] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:51.275956 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196cffeb-b85b-4744-9360-e51efdd6b797 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.280142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] MFA Rules not processed for user `aa8812ac328e45a8aae8f57d2a69efa3`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:51.281439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:51.352602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196cffeb-b85b-4744-9360-e51efdd6b797 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.354027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 873/1741] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:51.366611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.368076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:51.421107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.423489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.423707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.423985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.424625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.426564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.445824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59538d06-4cb5-48b3-8984-9af339ea74b7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.452837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.454177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.454466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.454745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.473948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.475779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.486008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 874/1742] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:51 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:51.502284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 869/1743] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:51 2022] POST /identity/v3/projects => generated 383 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:51.510271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.522583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72aa5337-e8bf-4021-bacc-6e66cd6730bf service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.554171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:51.605256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:51.624750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.625495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.625878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.626257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.626980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.629310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.735712 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 875/1744] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:51 2022] POST /identity/v3/users => generated 292 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:51.738460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.754943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d15bf7a-10ec-4bef-a092-0bd982d876b7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.754943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.756915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.756915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.756915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.757520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.758737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.768363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.769029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b387e-6582-43c8-9a78-56b1559c606e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.770163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 870/1745] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:51.774175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.774321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.775061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.775410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.775752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] PATH_INFO: `/v3/users/aa8812ac328e45a8aae8f57d2a69efa3` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.777019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa8812ac328e45a8aae8f57d2a69efa3)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.779561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.785429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fecfcce-759c-423e-9e49-877ff5bb773f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.790165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.794990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.794990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.794990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.794990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.794990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.804287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 871/1746] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:51.808391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:51.818737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb534f2-65f4-41ce-8e5d-b60b57bdba4d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:51.824224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337bc4ad-c3d7-4f95-8ede-142946bbbfd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:51.824963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337bc4ad-c3d7-4f95-8ede-142946bbbfd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:51.825378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337bc4ad-c3d7-4f95-8ede-142946bbbfd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:51.826292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337bc4ad-c3d7-4f95-8ede-142946bbbfd0 admin admin] PATH_INFO: `/v3/projects/cf60934225dc495b8d343dd83e9ffb93/users/b69951a29d3b4c8896a0f703f787f357/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:51.844226 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337bc4ad-c3d7-4f95-8ede-142946bbbfd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf60934225dc495b8d343dd83e9ffb93, user_id=b69951a29d3b4c8896a0f703f787f357, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:51.846961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337bc4ad-c3d7-4f95-8ede-142946bbbfd0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:51.956989 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 872/1747] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:51 2022] PUT /identity/v3/projects/cf60934225dc495b8d343dd83e9ffb93/users/b69951a29d3b4c8896a0f703f787f357/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:51.987017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa8812ac328e45a8aae8f57d2a69efa3'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:51.989142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa8812ac328e45a8aae8f57d2a69efa3'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:52.008287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa8812ac328e45a8aae8f57d2a69efa3'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:52.030827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.031121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.031407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.033697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa8812ac328e45a8aae8f57d2a69efa3'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:52.034829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1adcaa6-f547-4b02-8f5b-3bb9e337ca7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa8812ac328e45a8aae8f57d2a69efa3'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:52.062949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] MFA Rules not processed for user `b69951a29d3b4c8896a0f703f787f357`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:52.064168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:52.111209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 876/1748] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:51 2022] DELETE /identity/v3/users/aa8812ac328e45a8aae8f57d2a69efa3 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:52.115709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.116298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.116298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.129378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 873/1749] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:52.138657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.139972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:52.140522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:52.141186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:52.201820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:52.214070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 877/1750] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:52.214209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93312cc-2847-48d3-9036-26512785a6b5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:52.218922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:52.219539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.219821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.220096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.220723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:52.222899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:52.225018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.226550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:52.241603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:52.298950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 874/1751] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:52 2022] POST /identity/v3/projects => generated 385 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:52.300396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.300761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2058aefa-3172-4a0e-8418-36a2ff95d844 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:52.365388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:52.366138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.366502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.366858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.385658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107cd82e-2a56-4492-8703-5633794850af admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:52.392508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:52.393172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.393789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.394068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.394864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:52.397127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:52.448554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:52.450041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:52.460042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 878/1752] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:52.514052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 875/1753] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:52 2022] POST /identity/v3/users => generated 299 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:52.515817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.528038 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25cc07-e110-45bb-a10c-e8d8ed17d3f8 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:52.534296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7f73b7-1632-4218-ab4d-cafe315130b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:52.534907 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7f73b7-1632-4218-ab4d-cafe315130b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.535193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7f73b7-1632-4218-ab4d-cafe315130b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.535472 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7f73b7-1632-4218-ab4d-cafe315130b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.536521 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7f73b7-1632-4218-ab4d-cafe315130b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:52.538202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7f73b7-1632-4218-ab4d-cafe315130b8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:52.553352 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 879/1754] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:52.554524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.565396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597ddccf-3b55-4e8c-b0f2-8d64da66696d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:52.570212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:52.570871 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.574992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.574992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] PATH_INFO: `/v3/projects/20c8b83045d04322b3f9fd4e589db0fb/users/9f938fe9cdc045e3be521a53434d5a3c/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.587106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20c8b83045d04322b3f9fd4e589db0fb, user_id=9f938fe9cdc045e3be521a53434d5a3c, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:52.589944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:52.663115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12f945c-3033-4166-9136-4f1f2e2b75f3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.663593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12f945c-3033-4166-9136-4f1f2e2b75f3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.663958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12f945c-3033-4166-9136-4f1f2e2b75f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.670734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 876/1755] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:52 2022] PUT /identity/v3/projects/20c8b83045d04322b3f9fd4e589db0fb/users/9f938fe9cdc045e3be521a53434d5a3c/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:52.675217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.696700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b12f945c-3033-4166-9136-4f1f2e2b75f3 None None] MFA Rules not processed for user `14729da7649a4ab785ff87981968d342`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:52.698475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b12f945c-3033-4166-9136-4f1f2e2b75f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:52.767801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96610f92-6659-4841-8135-65a93a1f3fe0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:52.770770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:52.775378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:52.776409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.776836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.777164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.778108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:52.780072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:52.788815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 880/1756] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:52.797754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 877/1757] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:52.798236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.816302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5286842c-32eb-4ec7-b20a-982b170fae8e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:52.823063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:52.823749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.824038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.824977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] PATH_INFO: `/v3/projects/20c8b83045d04322b3f9fd4e589db0fb/users/9f938fe9cdc045e3be521a53434d5a3c/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.828661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] RBAC: Authorizing `identity:create_grant(project_id=20c8b83045d04322b3f9fd4e589db0fb, user_id=9f938fe9cdc045e3be521a53434d5a3c, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:52.834689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:52.850830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 878/1758] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:52 2022] PUT /identity/v3/projects/20c8b83045d04322b3f9fd4e589db0fb/users/9f938fe9cdc045e3be521a53434d5a3c/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:52.861306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:52.864037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:52.864037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:52.864037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:52.893382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] MFA Rules not processed for user `9f938fe9cdc045e3be521a53434d5a3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:52.894707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:52.913027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:52.914329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:53.012398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 881/1759] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:52 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:53.023235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:53.031648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:53.032889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdd3ef-b84b-48f8-9b94-7d2498310deb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.038583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.039351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.039744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.040096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] PATH_INFO: `/v3/projects/801236b29780486981d6f3105f61ea56` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.043311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=801236b29780486981d6f3105f61ea56)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:53.045581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:53.099711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:53.109549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb35b4f2-c323-4a7d-bf0c-611c3fc89d20 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.109549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe186f4-e79d-4b87-9a91-076746bbbf1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.109549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe186f4-e79d-4b87-9a91-076746bbbf1d service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.109549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe186f4-e79d-4b87-9a91-076746bbbf1d service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.109999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe186f4-e79d-4b87-9a91-076746bbbf1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.109999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bbe186f4-e79d-4b87-9a91-076746bbbf1d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:53.174326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '801236b29780486981d6f3105f61ea56', 'request_id': 'req-985cdde1-3914-4ed4-b6e4-167dec9004e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:53.177253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe186f4-e79d-4b87-9a91-076746bbbf1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:53.179267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe186f4-e79d-4b87-9a91-076746bbbf1d service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:53.194913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 882/1760] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:53.230813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '801236b29780486981d6f3105f61ea56', 'request_id': 'req-985cdde1-3914-4ed4-b6e4-167dec9004e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:53.241467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '801236b29780486981d6f3105f61ea56', 'request_id': 'req-985cdde1-3914-4ed4-b6e4-167dec9004e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:53.251443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] The token cache is being invalidate because project 801236b29780486981d6f3105f61ea56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:53.251744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:53.252457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-985cdde1-3914-4ed4-b6e4-167dec9004e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:53.254182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 879/1761] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:52 2022] DELETE /identity/v3/projects/801236b29780486981d6f3105f61ea56 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:53.259153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5356ef76-7699-47e0-a1ae-98d8755baab9 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.259436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5356ef76-7699-47e0-a1ae-98d8755baab9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.259709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5356ef76-7699-47e0-a1ae-98d8755baab9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.286863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5356ef76-7699-47e0-a1ae-98d8755baab9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:53.288760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5356ef76-7699-47e0-a1ae-98d8755baab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:53.301141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:53.385776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 883/1762] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:53.397098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d086e9-acd8-442d-a7c4-d8572d652425 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.397629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d086e9-acd8-442d-a7c4-d8572d652425 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.398098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d086e9-acd8-442d-a7c4-d8572d652425 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.430525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-89d086e9-acd8-442d-a7c4-d8572d652425 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:53.431820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-89d086e9-acd8-442d-a7c4-d8572d652425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:53.443266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 880/1763] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:53.446169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5356ef76-7699-47e0-a1ae-98d8755baab9 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:53.460987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5356ef76-7699-47e0-a1ae-98d8755baab9 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.465874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.466581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.466930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.467235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.468032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:53.468032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:53.470871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:53.487085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 884/1764] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:53.498162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d086e9-acd8-442d-a7c4-d8572d652425 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:53.500844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d086e9-acd8-442d-a7c4-d8572d652425 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.507570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.508370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.509020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.509474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.520928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:53.523025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:53.631518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 881/1765] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:53 2022] POST /identity/v3/projects => generated 373 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:53.641163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:53.666895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1e9723-fffe-436c-81f8-84a299273b4c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.761063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.761905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.762292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.762600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.764424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:53.766467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:53.871070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 885/1766] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:53 2022] POST /identity/v3/users => generated 287 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:53.875447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:53.885757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa334c-cce2-4eee-99b8-69490221a47f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.893612 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.906656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.907122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.907540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.908297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:53.910197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:53.921512 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 882/1767] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:53 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:53.925562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:53.938223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fce2e1d-a22d-4622-8a80-5c530584f3dd admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.943000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e920e1-49b6-4412-9bcb-db1823a561d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.943618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e920e1-49b6-4412-9bcb-db1823a561d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.943908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e920e1-49b6-4412-9bcb-db1823a561d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.944188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e920e1-49b6-4412-9bcb-db1823a561d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:53.944834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e920e1-49b6-4412-9bcb-db1823a561d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:53.946467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e920e1-49b6-4412-9bcb-db1823a561d6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:53.958505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 886/1768] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:53.963311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:53.977213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c3c54b-e24f-4820-b8e6-b3aa84a96d88 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:53.985073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:53.985829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:53.994582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:53.994582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] PATH_INFO: `/v3/projects/4ac3abcb19b745439c2fa02dc566fef2/users/9fb0a0cc698f4e0eac1d2e9cb2da8a04/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.004027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ac3abcb19b745439c2fa02dc566fef2, user_id=9fb0a0cc698f4e0eac1d2e9cb2da8a04, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.006863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.083133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 883/1769] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:53 2022] PUT /identity/v3/projects/4ac3abcb19b745439c2fa02dc566fef2/users/9fb0a0cc698f4e0eac1d2e9cb2da8a04/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:54.093821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a223af-b91e-47a8-a50b-336ee83fd1d3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.094144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a223af-b91e-47a8-a50b-336ee83fd1d3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.094425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a223af-b91e-47a8-a50b-336ee83fd1d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.126865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a1a223af-b91e-47a8-a50b-336ee83fd1d3 None None] MFA Rules not processed for user `9fb0a0cc698f4e0eac1d2e9cb2da8a04`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:54.128143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a1a223af-b91e-47a8-a50b-336ee83fd1d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:54.135588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.137503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:54.198276 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:54.212273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f67a3f-5c14-42e5-b14a-6104d2545427 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.217025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 887/1770] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:54.218564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a223af-b91e-47a8-a50b-336ee83fd1d3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.221121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.221866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.222149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.222430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] PATH_INFO: `/v3/users/34940012678441ac88b898dbd0d323db` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.223586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34940012678441ac88b898dbd0d323db)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.225654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.229779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a223af-b91e-47a8-a50b-336ee83fd1d3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.234571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.235336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.235692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.236040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.236757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.238912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.244608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:54.335661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 888/1771] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:54 2022] POST /identity/v3/projects => generated 373 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:54.339558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.402810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd226d-1de6-4daf-ae39-c662415056f2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.408259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.408924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.409193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.409502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.421575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.421575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.421785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34940012678441ac88b898dbd0d323db'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:54.430359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34940012678441ac88b898dbd0d323db'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:54.521847 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34940012678441ac88b898dbd0d323db'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:54.522528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34940012678441ac88b898dbd0d323db'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:54.523173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f967428-3158-456f-889e-b75e4cea6390 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34940012678441ac88b898dbd0d323db'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:54.529925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 889/1772] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:54 2022] POST /identity/v3/users => generated 293 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:54.533885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.535820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:54.540462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 884/1773] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:54 2022] DELETE /identity/v3/users/34940012678441ac88b898dbd0d323db => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:54.545384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.545789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.546172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.552799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f717909c-222d-422d-87fe-f9ad5acd4c76 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.566592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:54.567914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:54.616217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.617107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.617228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.617648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.618377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.620008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.634659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 890/1774] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:54 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:54.634834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 885/1775] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:54.638293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.642039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.643568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:54.648548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64267eca-aea7-4c12-880e-72c39eece4c4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.653179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.654012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.654304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.654577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] PATH_INFO: `/v3/projects/07d5ce8a62dc4c33a890f1cdc6cc9856/users/9d110106608847608b03a0342af28aeb/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.675953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07d5ce8a62dc4c33a890f1cdc6cc9856, user_id=9d110106608847608b03a0342af28aeb, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.679210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.702779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3550cf8-2e7f-4baf-a448-9770ad451a95 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.706406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.707016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.707298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.710743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.719505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.721148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.732654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 886/1776] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:54.750791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 891/1777] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:54 2022] PUT /identity/v3/projects/07d5ce8a62dc4c33a890f1cdc6cc9856/users/9d110106608847608b03a0342af28aeb/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:54.754489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.767328 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7525745-2392-4fcd-a55e-e9553a2184e7 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.781738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:54.839004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7002effb-3d0d-4921-bb59-5fc7772710be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.839781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7002effb-3d0d-4921-bb59-5fc7772710be admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.840181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7002effb-3d0d-4921-bb59-5fc7772710be admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.840526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7002effb-3d0d-4921-bb59-5fc7772710be admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.841752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7002effb-3d0d-4921-bb59-5fc7772710be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.843644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7002effb-3d0d-4921-bb59-5fc7772710be admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.854993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 887/1778] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:54 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:54.859283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:54.870083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51bacae-5367-459f-add8-2efd244caf63 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:54.875451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:54.876115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.876395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.876670 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] PATH_INFO: `/v3/projects/07d5ce8a62dc4c33a890f1cdc6cc9856/users/9d110106608847608b03a0342af28aeb/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.878590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07d5ce8a62dc4c33a890f1cdc6cc9856, user_id=9d110106608847608b03a0342af28aeb, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:54.881204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:54.893146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 892/1779] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:54 2022] PUT /identity/v3/projects/07d5ce8a62dc4c33a890f1cdc6cc9856/users/9d110106608847608b03a0342af28aeb/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:06:54.903547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:54.903922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:54.904218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:54.933085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] MFA Rules not processed for user `9d110106608847608b03a0342af28aeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:54.934382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:55.041944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 888/1780] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:54 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:55.056417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:55.058674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:55.108565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:55.129860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65005bef-4a69-430d-bcd7-df9d4a824ec6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:55.134287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:55.135007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:55.135362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:55.135740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:55.147456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:55.149107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:55.159845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 893/1781] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:55 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:55.249946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:55.252097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:55.329098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac78f95-e5c8-4590-8ebe-0b4edaa7aa45 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:55.334386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:55.335031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:55.335318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:55.335599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] PATH_INFO: `/v3/projects/bbc0a39dd74649989d17306408e4b90c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:55.336756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] RBAC: Authorizing `identity:delete_project(project_id=bbc0a39dd74649989d17306408e4b90c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:55.338692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:55.390571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:55.401650 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18ada31-4461-47fc-b8b2-ac7fc6f13e93 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:55.407308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:55.407956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:55.411934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:55.411934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:55.411934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:55.458420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bbc0a39dd74649989d17306408e4b90c', 'request_id': 'req-465a3007-4848-4c7b-b652-daffeb97a77c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.477441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:55.478871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:55.488819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 894/1782] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:55.489771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:55.491198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:55.502281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33d0f3e-b1d6-4665-8477-635be3759d50 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:55.544965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bbc0a39dd74649989d17306408e4b90c', 'request_id': 'req-465a3007-4848-4c7b-b652-daffeb97a77c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.552692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bbc0a39dd74649989d17306408e4b90c', 'request_id': 'req-465a3007-4848-4c7b-b652-daffeb97a77c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.558115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:55.558854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:55.559143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:55.559547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:55.561964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:55.564921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] The token cache is being invalidate because project bbc0a39dd74649989d17306408e4b90c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:55.565282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.566261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.567822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 889/1783] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:55 2022] DELETE /identity/v3/projects/bbc0a39dd74649989d17306408e4b90c => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:55.582866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:55.585107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:55.633720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:55.635226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:55.636503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:55.645675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 895/1784] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:55 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:55.661369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465a3007-4848-4c7b-b652-daffeb97a77c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:55.666692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:55.667367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:55.667646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:55.667939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] PATH_INFO: `/v3/users/a569ec01c5a44f84857af4bff8ffb94c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:55.669744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a569ec01c5a44f84857af4bff8ffb94c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:55.671667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:55.845589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a569ec01c5a44f84857af4bff8ffb94c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.858120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a569ec01c5a44f84857af4bff8ffb94c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.973831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a569ec01c5a44f84857af4bff8ffb94c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.975357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a569ec01c5a44f84857af4bff8ffb94c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.976173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-63773d0f-227e-4331-a098-cd609966a10a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a569ec01c5a44f84857af4bff8ffb94c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:55.997100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 890/1785] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:06:55 2022] DELETE /identity/v3/users/a569ec01c5a44f84857af4bff8ffb94c => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:56.553283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:56.565731 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456cc67f-48eb-4a37-8a42-a3396199eeef service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:56.605302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:56.639622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:56.640328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:56.640605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:56.640915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] PATH_INFO: `/v3/projects/63c2b251ad8e48c89e025c59c7f61cca` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:56.642090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63c2b251ad8e48c89e025c59c7f61cca)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:56.644045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:56.738719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63c2b251ad8e48c89e025c59c7f61cca', 'request_id': 'req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:56.744729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63c2b251ad8e48c89e025c59c7f61cca', 'request_id': 'req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:56.753165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63c2b251ad8e48c89e025c59c7f61cca', 'request_id': 'req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:56.831668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] The token cache is being invalidate because project 63c2b251ad8e48c89e025c59c7f61cca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:06:56.831953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:56.832882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-51d7e21d-6da0-4e0d-97c9-b12f168d22f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:06:56.834541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 896/1786] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:06:56 2022] DELETE /identity/v3/projects/63c2b251ad8e48c89e025c59c7f61cca => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:56.839443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50bb945-9843-4c44-b859-9255b65777dd None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:56.839916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50bb945-9843-4c44-b859-9255b65777dd None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:56.840017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50bb945-9843-4c44-b859-9255b65777dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:56.872791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a50bb945-9843-4c44-b859-9255b65777dd None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:56.875100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a50bb945-9843-4c44-b859-9255b65777dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:56.921561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:56.940038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 891/1787] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:56.949639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f823cb4-434d-4ccf-a601-a07396ad6adc None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:56.949923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f823cb4-434d-4ccf-a601-a07396ad6adc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:56.950271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f823cb4-434d-4ccf-a601-a07396ad6adc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:56.972135 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-7f823cb4-434d-4ccf-a601-a07396ad6adc None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:56.973527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7f823cb4-434d-4ccf-a601-a07396ad6adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:56.983423 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 897/1788] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:06:56.987688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50bb945-9843-4c44-b859-9255b65777dd None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.004213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50bb945-9843-4c44-b859-9255b65777dd None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.016344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.018056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.018725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.018982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.019738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:06:57.020045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.021838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.049479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 892/1789] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:06:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:06:57.050708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f823cb4-434d-4ccf-a601-a07396ad6adc None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.061189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f823cb4-434d-4ccf-a601-a07396ad6adc None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.066445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.067076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.067519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.067807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.068503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.070719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.180261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 898/1790] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:57 2022] POST /identity/v3/projects => generated 407 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:57.185050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.197349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82e5f0d-9ed9-4eec-a3e4-c682c7d9f7e2 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.312860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.313564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.313857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.314146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.315179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.316743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.359600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.361740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:57.435588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868a7672-31e4-4b88-be3c-93bcf82568ef admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.439282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d86b17-ebd9-44df-8cb4-e74ceb272bfa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.439889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d86b17-ebd9-44df-8cb4-e74ceb272bfa service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.440174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d86b17-ebd9-44df-8cb4-e74ceb272bfa service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.440466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d86b17-ebd9-44df-8cb4-e74ceb272bfa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.442249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-00d86b17-ebd9-44df-8cb4-e74ceb272bfa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:57.443867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 893/1791] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:57 2022] POST /identity/v3/users => generated 304 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:57.448079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.468866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ee0a52-9727-4350-9e5d-9642c111b302 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.474695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.475555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.475843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.476123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.477007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.479138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.491600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 894/1792] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:57.495426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.496676 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:57.509833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72961436-a1d0-48ff-ace6-67ba7ece3a86 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.517929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.517929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.517929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.517929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.518264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.519830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.527352 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d86b17-ebd9-44df-8cb4-e74ceb272bfa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.531386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d86b17-ebd9-44df-8cb4-e74ceb272bfa service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.536731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 895/1793] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:57.543813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.554049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 899/1794] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:57 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:06:57.561571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd1b87f-efdd-4b86-9779-8e7d369d741c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.567002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.567770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.568062 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.568343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] PATH_INFO: `/v3/projects/0177d47dde7345faa4b63b1fa44287c6/users/e88e11125c7b497293f296c93a536b7e/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.589233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0177d47dde7345faa4b63b1fa44287c6, user_id=e88e11125c7b497293f296c93a536b7e, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.592765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.660155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 896/1795] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:57 2022] PUT /identity/v3/projects/0177d47dde7345faa4b63b1fa44287c6/users/e88e11125c7b497293f296c93a536b7e/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:57.670596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbf2352-1081-4103-ac8f-e64fe7c47db4 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.671164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbf2352-1081-4103-ac8f-e64fe7c47db4 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.671613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbf2352-1081-4103-ac8f-e64fe7c47db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.709705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3dbf2352-1081-4103-ac8f-e64fe7c47db4 None None] MFA Rules not processed for user `e88e11125c7b497293f296c93a536b7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:57.711063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3dbf2352-1081-4103-ac8f-e64fe7c47db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:57.798310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 900/1796] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:57 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:57.803180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:57.887856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84c006f-32ff-400d-b440-c4db6959098b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:57.895356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:57.896039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:57.896334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:57.896615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:57.897344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:57.899275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:57.997983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 897/1797] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:06:57 2022] POST /identity/v3/projects => generated 409 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:06:58.002107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbf2352-1081-4103-ac8f-e64fe7c47db4 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:58.020874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbf2352-1081-4103-ac8f-e64fe7c47db4 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:58.085601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:58.086227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:58.086602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:58.086783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:58.087399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:58.089332 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:58.159151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 901/1798] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:06:58 2022] POST /identity/v3/users => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:06:58.163281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:58.171744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:58.181344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dface49a-4335-42af-a839-301c0628d04d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:58.187497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:58.188180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:58.188503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:58.188733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:58.189410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:58.191245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:58.204196 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 898/1799] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:58 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:06:58.209303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:58.221612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b2a4a7-e24e-4ac8-89e9-90c6fefc845a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:58.228999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:58.230055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:58.230351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:58.230634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] PATH_INFO: `/v3/projects/0aa78c0bc4b747a2a30b33efec7f34ba/users/a47ee02ef5ce458bbef9beecce39a675/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:58.248764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0aa78c0bc4b747a2a30b33efec7f34ba, user_id=a47ee02ef5ce458bbef9beecce39a675, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:58.251744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:58.299871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 902/1800] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:58 2022] PUT /identity/v3/projects/0aa78c0bc4b747a2a30b33efec7f34ba/users/a47ee02ef5ce458bbef9beecce39a675/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:58.304174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:58.315402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c5902e-c4d5-4e9e-b34e-f88cb921799b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:58.378266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db240e63-657e-40f7-beb5-b5eba184330a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:58.378864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db240e63-657e-40f7-beb5-b5eba184330a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:58.379144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db240e63-657e-40f7-beb5-b5eba184330a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:58.379415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db240e63-657e-40f7-beb5-b5eba184330a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:58.380044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db240e63-657e-40f7-beb5-b5eba184330a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:58.381666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db240e63-657e-40f7-beb5-b5eba184330a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:58.392901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 899/1801] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:06:58 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:06:58.396146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:58.406620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbf7689-411a-4df8-9a6d-0c1201661f31 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:58.411316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:58.411973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:58.412260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:58.412539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] PATH_INFO: `/v3/projects/0aa78c0bc4b747a2a30b33efec7f34ba/users/a47ee02ef5ce458bbef9beecce39a675/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:58.414463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0aa78c0bc4b747a2a30b33efec7f34ba, user_id=a47ee02ef5ce458bbef9beecce39a675, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:58.417109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:58.428482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 903/1802] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:06:58 2022] PUT /identity/v3/projects/0aa78c0bc4b747a2a30b33efec7f34ba/users/a47ee02ef5ce458bbef9beecce39a675/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:06:58.439352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53102659-80b7-4eb4-be26-994add5fdab4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:58.439641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53102659-80b7-4eb4-be26-994add5fdab4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:58.439915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53102659-80b7-4eb4-be26-994add5fdab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:58.470805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-53102659-80b7-4eb4-be26-994add5fdab4 None None] MFA Rules not processed for user `a47ee02ef5ce458bbef9beecce39a675`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:06:58.472283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-53102659-80b7-4eb4-be26-994add5fdab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:58.574799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 900/1803] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:06:58 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:06:58.588692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:58.593694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:06:58.647202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:06:58.679214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf9edb-bed8-4c79-96e0-176db414b11b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:58.683937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:58.684982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:58.685373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:58.685712 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:58.695861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:58.697336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:58.707293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 904/1804] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:58 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:06:59.210422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53102659-80b7-4eb4-be26-994add5fdab4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:06:59.224878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53102659-80b7-4eb4-be26-994add5fdab4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:06:59.229671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9653c7f-fe61-42cf-bfd0-278e71dbb0c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:06:59.230516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9653c7f-fe61-42cf-bfd0-278e71dbb0c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:06:59.230864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9653c7f-fe61-42cf-bfd0-278e71dbb0c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:06:59.231211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9653c7f-fe61-42cf-bfd0-278e71dbb0c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:06:59.294799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9653c7f-fe61-42cf-bfd0-278e71dbb0c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:06:59.297484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9653c7f-fe61-42cf-bfd0-278e71dbb0c4 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:06:59.310271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 901/1805] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:06:59 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:07:03.738319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:03.739879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:03.806605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae867bd1-8833-4c93-8cca-712c516177a1 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:03.812398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:03.813034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:03.813313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:03.813614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] PATH_INFO: `/v3/users/3a71a3f992314746a56dee324c1f111a` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:03.814750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a71a3f992314746a56dee324c1f111a)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:03.816903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:04.057446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a71a3f992314746a56dee324c1f111a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:04.057446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a71a3f992314746a56dee324c1f111a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:04.073514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a71a3f992314746a56dee324c1f111a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:04.087529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a71a3f992314746a56dee324c1f111a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:04.088288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a71a3f992314746a56dee324c1f111a'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:04.187421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 905/1806] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:03 2022] DELETE /identity/v3/users/3a71a3f992314746a56dee324c1f111a => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:07:04.190755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:04.191727 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:04.192231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:04.213227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:04.214489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:04.220200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:04.274913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 902/1807] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:04.284595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:04.286025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:04.344388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:04.349595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804291bb-55e4-4429-9a79-0e8887b32b1d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:04.353787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:04.354421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:04.354707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:04.354987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:04.364855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:04.366337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:04.376651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 906/1808] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:07:04.786898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:04.788400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:04.815547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9250a2-21a0-4a3f-b8ec-c6d0a300afc5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:04.815547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:04.815547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:04.815547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:04.815913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] PATH_INFO: `/v3/projects/e0b7ca901b75407ea76ec658ca4f5c70` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:04.815913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0b7ca901b75407ea76ec658ca4f5c70)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:04.815913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:05.002799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0b7ca901b75407ea76ec658ca4f5c70', 'request_id': 'req-97f8b796-983e-4e5a-b604-05c232cf74a5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.104627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0b7ca901b75407ea76ec658ca4f5c70', 'request_id': 'req-97f8b796-983e-4e5a-b604-05c232cf74a5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.113299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0b7ca901b75407ea76ec658ca4f5c70', 'request_id': 'req-97f8b796-983e-4e5a-b604-05c232cf74a5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.125264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] The token cache is being invalidate because project e0b7ca901b75407ea76ec658ca4f5c70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:07:05.125587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.126302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.127853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 903/1809] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:07:04 2022] DELETE /identity/v3/projects/e0b7ca901b75407ea76ec658ca4f5c70 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:05.132181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:05.142663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348329a7-8010-4454-969b-407e1691ca9e service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:05.199118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:05.199570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:05.199855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:05.200281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] PATH_INFO: `/v3/users/b69951a29d3b4c8896a0f703f787f357` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:05.202223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b69951a29d3b4c8896a0f703f787f357)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:05.204193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:05.399280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b69951a29d3b4c8896a0f703f787f357'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.402339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b69951a29d3b4c8896a0f703f787f357'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.413020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b69951a29d3b4c8896a0f703f787f357'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.421382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b69951a29d3b4c8896a0f703f787f357'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.422798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b69951a29d3b4c8896a0f703f787f357'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:05.578911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 907/1810] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:05 2022] DELETE /identity/v3/users/b69951a29d3b4c8896a0f703f787f357 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:06.049291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:06.052166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:06.067087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f8b796-983e-4e5a-b604-05c232cf74a5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:06.128126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:06.128753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:06.129037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:06.129339 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] PATH_INFO: `/v3/projects/cf60934225dc495b8d343dd83e9ffb93` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:06.130454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf60934225dc495b8d343dd83e9ffb93)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:06.133415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:06.227706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:06.237517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf60934225dc495b8d343dd83e9ffb93', 'request_id': 'req-742303ae-2c2c-46d3-acef-4a996a66b626'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.367244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf60934225dc495b8d343dd83e9ffb93', 'request_id': 'req-742303ae-2c2c-46d3-acef-4a996a66b626'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.374466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf60934225dc495b8d343dd83e9ffb93', 'request_id': 'req-742303ae-2c2c-46d3-acef-4a996a66b626'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.385143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] The token cache is being invalidate because project cf60934225dc495b8d343dd83e9ffb93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:07:06.385471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.386109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.387598 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 904/1811] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:07:06 2022] DELETE /identity/v3/projects/cf60934225dc495b8d343dd83e9ffb93 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:06.392101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:06.393574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:06.407133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763f948d-3874-481f-8c55-bb6c1cc9194a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:06.473682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:06.476336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:06.476336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:06.476336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] PATH_INFO: `/v3/users/9f938fe9cdc045e3be521a53434d5a3c` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:06.476336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f938fe9cdc045e3be521a53434d5a3c)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:06.478034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:06.490515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:06.678631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f938fe9cdc045e3be521a53434d5a3c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.679224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f938fe9cdc045e3be521a53434d5a3c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.691540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f938fe9cdc045e3be521a53434d5a3c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.698346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f938fe9cdc045e3be521a53434d5a3c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.699005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f158fbd-d6bb-41e3-9ce2-4a01e7be02a4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f938fe9cdc045e3be521a53434d5a3c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:06.802028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 908/1812] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:06 2022] DELETE /identity/v3/users/9f938fe9cdc045e3be521a53434d5a3c => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:07.192994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:07.194502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:07.208453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742303ae-2c2c-46d3-acef-4a996a66b626 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:07.266426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:07.267066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.267350 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.267624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] PATH_INFO: `/v3/projects/20c8b83045d04322b3f9fd4e589db0fb` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.268737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=20c8b83045d04322b3f9fd4e589db0fb)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:07.270705 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:07.374168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20c8b83045d04322b3f9fd4e589db0fb', 'request_id': 'req-fe62de41-43d2-46b4-894b-5b2105a02d5c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:07.447210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20c8b83045d04322b3f9fd4e589db0fb', 'request_id': 'req-fe62de41-43d2-46b4-894b-5b2105a02d5c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:07.454470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20c8b83045d04322b3f9fd4e589db0fb', 'request_id': 'req-fe62de41-43d2-46b4-894b-5b2105a02d5c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:07.466113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] The token cache is being invalidate because project 20c8b83045d04322b3f9fd4e589db0fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:07:07.466418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:07.467690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fe62de41-43d2-46b4-894b-5b2105a02d5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:07.469273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 905/1813] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:07:07 2022] DELETE /identity/v3/projects/20c8b83045d04322b3f9fd4e589db0fb => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:07.474414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bfe5fd-46dc-40ed-b69e-24c25259b43d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.474869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bfe5fd-46dc-40ed-b69e-24c25259b43d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.474958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bfe5fd-46dc-40ed-b69e-24c25259b43d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.495128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-83bfe5fd-46dc-40ed-b69e-24c25259b43d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:07.496372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-83bfe5fd-46dc-40ed-b69e-24c25259b43d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:07.556840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 909/1814] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:07.566367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170aa9a-e634-4587-8108-00230a5fef6a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.566367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170aa9a-e634-4587-8108-00230a5fef6a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.566367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170aa9a-e634-4587-8108-00230a5fef6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.596234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b170aa9a-e634-4587-8108-00230a5fef6a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:07.597502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b170aa9a-e634-4587-8108-00230a5fef6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:07.605803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 906/1815] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:07.611692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bfe5fd-46dc-40ed-b69e-24c25259b43d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:07.621622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bfe5fd-46dc-40ed-b69e-24c25259b43d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:07.626628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:07.627249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.627508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.627778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.628420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:07:07.628718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:07.630333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:07.642486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 910/1816] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:07:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:07:07.646879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b170aa9a-e634-4587-8108-00230a5fef6a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:07.657661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b170aa9a-e634-4587-8108-00230a5fef6a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:07.662460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:07.663085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.663371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.663649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.664296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:07.666301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:07.763109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 907/1817] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:07:07 2022] POST /identity/v3/projects => generated 391 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:07:07.767652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:07.777801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf34e58-74bb-4f3c-8f7d-f53440375cfd admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:07.835922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:07.836535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.836821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.837100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.837761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:07.839678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:07.904060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 911/1818] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:07:07 2022] POST /identity/v3/users => generated 296 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:07:07.908478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:07.920298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715e37cb-a0b7-4b49-aa91-1e0fc646a806 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:07.925130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:07.925815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.926099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.926374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.927009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:07.928592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:07.939304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 908/1819] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:07.943134 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:07.949698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:07.953596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9971617-372a-47ac-b9c9-11ac059b0488 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:07.958504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a34814-87cc-413b-b8e7-8bb07d17ef37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:07.961834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a34814-87cc-413b-b8e7-8bb07d17ef37 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.961834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a34814-87cc-413b-b8e7-8bb07d17ef37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.961834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a34814-87cc-413b-b8e7-8bb07d17ef37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:07.961834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a34814-87cc-413b-b8e7-8bb07d17ef37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:07.962193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a34814-87cc-413b-b8e7-8bb07d17ef37 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:07.972621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 912/1820] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:07.976578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:07.988086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54528b26-7140-4ccb-97f5-fa973d7dd324 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:07.992841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:07.993581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:07.993813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:07.994166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] PATH_INFO: `/v3/projects/fa88a1f80f5f4af699682e7d4c4a3197/users/6f9801053b5d45269d940c757f24b65d/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:08.018688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:08.024467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa88a1f80f5f4af699682e7d4c4a3197, user_id=6f9801053b5d45269d940c757f24b65d, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:08.034524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:08.104864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 909/1821] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:07:07 2022] PUT /identity/v3/projects/fa88a1f80f5f4af699682e7d4c4a3197/users/6f9801053b5d45269d940c757f24b65d/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:08.116579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:08.116873 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:08.117150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:08.146896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] MFA Rules not processed for user `6f9801053b5d45269d940c757f24b65d`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:08.148101 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:08.214728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 913/1822] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:08 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:08.226531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:08.228217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:08.291076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a273a19-99ad-4688-b0a1-176b1986f2c9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:08.296398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e645d3-51a2-4aa9-9f83-48a855276cd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:08.297025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e645d3-51a2-4aa9-9f83-48a855276cd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:08.297300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e645d3-51a2-4aa9-9f83-48a855276cd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:08.297887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e645d3-51a2-4aa9-9f83-48a855276cd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:08.307265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e645d3-51a2-4aa9-9f83-48a855276cd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:08.308726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e645d3-51a2-4aa9-9f83-48a855276cd3 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:08.319371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 910/1823] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:08 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:07:15.352406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:15.353983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:15.435292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0680088-c290-4690-90a2-ea3d71c10782 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:15.441258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:15.441938 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:15.447069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:15.447069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] PATH_INFO: `/v3/users/e88e11125c7b497293f296c93a536b7e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:15.447069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e88e11125c7b497293f296c93a536b7e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:15.447069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:15.658788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e88e11125c7b497293f296c93a536b7e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:15.660184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e88e11125c7b497293f296c93a536b7e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:15.676673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e88e11125c7b497293f296c93a536b7e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:15.683655 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e88e11125c7b497293f296c93a536b7e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:15.684371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e88e11125c7b497293f296c93a536b7e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:15.750755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 914/1824] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:15 2022] DELETE /identity/v3/users/e88e11125c7b497293f296c93a536b7e => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:15.755379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:15.755878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:15.755972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:15.776592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:15.778554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:15.838333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 911/1825] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:15.848487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:15.850011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:15.855622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:15.907612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e530d0-32d4-47b5-81e7-8e5dcf05aeb5 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:15.911305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:15.911913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:15.912193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:15.912466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:15.921970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:15.926657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:15.933074 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 915/1826] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:07:16.397970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:16.399385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:16.416119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d3975e-3e25-4b74-8548-33bda55628e4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:16.422045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:16.422814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:16.423158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:16.423510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] PATH_INFO: `/v3/projects/0177d47dde7345faa4b63b1fa44287c6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:16.425596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0177d47dde7345faa4b63b1fa44287c6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:16.430334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:16.533969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0177d47dde7345faa4b63b1fa44287c6', 'request_id': 'req-ff0615eb-f1dd-4310-9838-edee1b191f55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.595798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0177d47dde7345faa4b63b1fa44287c6', 'request_id': 'req-ff0615eb-f1dd-4310-9838-edee1b191f55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.603497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0177d47dde7345faa4b63b1fa44287c6', 'request_id': 'req-ff0615eb-f1dd-4310-9838-edee1b191f55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.616545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] The token cache is being invalidate because project 0177d47dde7345faa4b63b1fa44287c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:07:16.616848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.617546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.619229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 912/1827] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:07:16 2022] DELETE /identity/v3/projects/0177d47dde7345faa4b63b1fa44287c6 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:16.623836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:16.634236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3a7659-8d48-47f6-832e-08153ad1a192 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:16.699168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:16.699839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:16.700122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:16.700404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] PATH_INFO: `/v3/users/a47ee02ef5ce458bbef9beecce39a675` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:16.702410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a47ee02ef5ce458bbef9beecce39a675)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:16.704390 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:16.930079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a47ee02ef5ce458bbef9beecce39a675'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.930684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a47ee02ef5ce458bbef9beecce39a675'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.943932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a47ee02ef5ce458bbef9beecce39a675'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.951859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a47ee02ef5ce458bbef9beecce39a675'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:16.952945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70e13419-0323-4263-a21d-ac0fd2b86f55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a47ee02ef5ce458bbef9beecce39a675'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:17.047648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 916/1828] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:16 2022] DELETE /identity/v3/users/a47ee02ef5ce458bbef9beecce39a675 => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:17.511577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:17.513130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:17.526907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0615eb-f1dd-4310-9838-edee1b191f55 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:17.540472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:17.585719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:17.586363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:17.586649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:17.586951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] PATH_INFO: `/v3/projects/0aa78c0bc4b747a2a30b33efec7f34ba` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:17.588041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0aa78c0bc4b747a2a30b33efec7f34ba)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:17.590044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:17.722681 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0aa78c0bc4b747a2a30b33efec7f34ba', 'request_id': 'req-c247a116-4f98-4b0b-b2a0-cfc327050a31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:17.778754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0aa78c0bc4b747a2a30b33efec7f34ba', 'request_id': 'req-c247a116-4f98-4b0b-b2a0-cfc327050a31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:17.786312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0aa78c0bc4b747a2a30b33efec7f34ba', 'request_id': 'req-c247a116-4f98-4b0b-b2a0-cfc327050a31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:17.797926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] The token cache is being invalidate because project 0aa78c0bc4b747a2a30b33efec7f34ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:07:17.798221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:17.798872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c247a116-4f98-4b0b-b2a0-cfc327050a31 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:17.800378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 913/1829] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:07:17 2022] DELETE /identity/v3/projects/0aa78c0bc4b747a2a30b33efec7f34ba => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:17.806281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c60664a-a10b-402f-b5a0-25228f99151d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:17.806763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c60664a-a10b-402f-b5a0-25228f99151d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:17.806863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c60664a-a10b-402f-b5a0-25228f99151d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:17.828684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-7c60664a-a10b-402f-b5a0-25228f99151d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:17.829977 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7c60664a-a10b-402f-b5a0-25228f99151d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:17.849227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:17.890857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 917/1830] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:17.898820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497e53f1-dfe5-4001-a2fd-46ccf767de55 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:17.899114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497e53f1-dfe5-4001-a2fd-46ccf767de55 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:17.899442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497e53f1-dfe5-4001-a2fd-46ccf767de55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:17.920333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-497e53f1-dfe5-4001-a2fd-46ccf767de55 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:17.921768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-497e53f1-dfe5-4001-a2fd-46ccf767de55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:17.929697 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 914/1831] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:07:17.934082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c60664a-a10b-402f-b5a0-25228f99151d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:17.944125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c60664a-a10b-402f-b5a0-25228f99151d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:17.949010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:17.952083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:17.952083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:17.952083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:17.952083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:07:17.952083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:17.953550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:17.965536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 918/1832] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:07:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:07:17.970530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497e53f1-dfe5-4001-a2fd-46ccf767de55 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:17.979700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497e53f1-dfe5-4001-a2fd-46ccf767de55 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:17.985207 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:17.985885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:17.986167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:17.986444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:17.987074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:17.988979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:18.101009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 915/1833] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:07:17 2022] POST /identity/v3/projects => generated 361 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:07:18.105241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:18.115997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafd502c-ce94-4537-b793-aac5a332b7c3 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:18.174892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:18.175508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:18.175786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:18.176060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:18.176695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:18.178701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:18.258615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 919/1834] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:07:18 2022] POST /identity/v3/users => generated 281 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:07:18.258800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:18.270385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f42866c-9b77-4c8e-97a4-b4569b632072 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:18.276108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46248334-d379-4c81-800f-24392377552e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:18.276926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46248334-d379-4c81-800f-24392377552e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:18.277272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46248334-d379-4c81-800f-24392377552e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:18.285492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46248334-d379-4c81-800f-24392377552e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:18.285492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46248334-d379-4c81-800f-24392377552e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:18.285492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46248334-d379-4c81-800f-24392377552e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:18.291047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 916/1835] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:18.295355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:18.306912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536006fe-d248-421a-9eb3-4a0f46997366 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:18.312469 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99de2b90-794c-4bba-a247-8977336812d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:18.313072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de2b90-794c-4bba-a247-8977336812d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:18.313393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de2b90-794c-4bba-a247-8977336812d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:18.313692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de2b90-794c-4bba-a247-8977336812d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:18.314312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99de2b90-794c-4bba-a247-8977336812d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:18.315915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99de2b90-794c-4bba-a247-8977336812d5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:18.326837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 920/1836] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:18.330716 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46248334-d379-4c81-800f-24392377552e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:18.340820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46248334-d379-4c81-800f-24392377552e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:18.345964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:18.346620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:18.346908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:18.347215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] PATH_INFO: `/v3/projects/d9489f0c270c400ba17c9affc5dec9e9/users/c609c047e5e745a09f92c6b0a9aa322d/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:18.363333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9489f0c270c400ba17c9affc5dec9e9, user_id=c609c047e5e745a09f92c6b0a9aa322d, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:18.366008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:18.447052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 917/1837] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:07:18 2022] PUT /identity/v3/projects/d9489f0c270c400ba17c9affc5dec9e9/users/c609c047e5e745a09f92c6b0a9aa322d/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:18.457414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5921804c-ff6e-43ed-b8b3-eb9b13739eef None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:18.457706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5921804c-ff6e-43ed-b8b3-eb9b13739eef None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:18.458005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5921804c-ff6e-43ed-b8b3-eb9b13739eef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:18.487424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5921804c-ff6e-43ed-b8b3-eb9b13739eef None None] MFA Rules not processed for user `c609c047e5e745a09f92c6b0a9aa322d`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:18.488633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5921804c-ff6e-43ed-b8b3-eb9b13739eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:18.552810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 921/1838] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:18 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:18.589304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:18.591047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:18.639222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:18.649427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183f10cc-6946-40d0-a985-bb9f8c96c7fa admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:18.653137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca606c4-cc4d-45db-8f59-55437127cc42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:18.653780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca606c4-cc4d-45db-8f59-55437127cc42 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:18.654608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca606c4-cc4d-45db-8f59-55437127cc42 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:18.654881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca606c4-cc4d-45db-8f59-55437127cc42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:18.663112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca606c4-cc4d-45db-8f59-55437127cc42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:18.664614 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca606c4-cc4d-45db-8f59-55437127cc42 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:18.673992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 918/1839] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:18 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:07:18.691016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5921804c-ff6e-43ed-b8b3-eb9b13739eef None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:18.712299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:18.755673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5921804c-ff6e-43ed-b8b3-eb9b13739eef None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:18.760522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:18.761183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:18.761499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:18.761774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] PATH_INFO: `/v3/users/c609c047e5e745a09f92c6b0a9aa322d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:18.762992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=c609c047e5e745a09f92c6b0a9aa322d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:18.765603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:18.943212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c609c047e5e745a09f92c6b0a9aa322d'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:18.944319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c609c047e5e745a09f92c6b0a9aa322d'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:18.957283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c609c047e5e745a09f92c6b0a9aa322d'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:18.965008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c609c047e5e745a09f92c6b0a9aa322d'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:18.965862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c609c047e5e745a09f92c6b0a9aa322d'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:19.019275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 922/1840] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:18 2022] DELETE /identity/v3/users/c609c047e5e745a09f92c6b0a9aa322d => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:19.023942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:19.024266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:19.024768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:19.050195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:19.051490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:19.113042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 919/1841] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:19.122892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:19.124485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:19.189923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061db162-5be5-40a4-b57f-c134a03335ee admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:19.194327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bbf44d-c8cd-459a-a218-cb3bf30de8a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:19.195027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bbf44d-c8cd-459a-a218-cb3bf30de8a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:19.195344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bbf44d-c8cd-459a-a218-cb3bf30de8a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:19.195647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bbf44d-c8cd-459a-a218-cb3bf30de8a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:19.208968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bbf44d-c8cd-459a-a218-cb3bf30de8a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:19.213257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bbf44d-c8cd-459a-a218-cb3bf30de8a9 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:19.221249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 923/1842] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:07:19.684453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:19.688227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:19.703284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb313de-5e13-4b3c-a1c6-6a205fb6bad3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:19.708638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:19.712075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:19.712075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:19.712075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] PATH_INFO: `/v3/projects/d9489f0c270c400ba17c9affc5dec9e9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:19.712075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9489f0c270c400ba17c9affc5dec9e9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:19.713064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:19.846241 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9489f0c270c400ba17c9affc5dec9e9', 'request_id': 'req-bff643a6-e567-4bd0-ab4e-9af9710fa053'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:20.001891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9489f0c270c400ba17c9affc5dec9e9', 'request_id': 'req-bff643a6-e567-4bd0-ab4e-9af9710fa053'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:20.009239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9489f0c270c400ba17c9affc5dec9e9', 'request_id': 'req-bff643a6-e567-4bd0-ab4e-9af9710fa053'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:20.034871 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] The token cache is being invalidate because project d9489f0c270c400ba17c9affc5dec9e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:07:20.035174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:20.035861 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bff643a6-e567-4bd0-ab4e-9af9710fa053 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:20.037432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 920/1843] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:07:19 2022] DELETE /identity/v3/projects/d9489f0c270c400ba17c9affc5dec9e9 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:20.042582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7037dbe-f39c-4da3-8c5e-c8341235231b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.043080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7037dbe-f39c-4da3-8c5e-c8341235231b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.043178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7037dbe-f39c-4da3-8c5e-c8341235231b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.065554 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-f7037dbe-f39c-4da3-8c5e-c8341235231b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:20.066805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f7037dbe-f39c-4da3-8c5e-c8341235231b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:20.107579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:20.133269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 924/1844] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:07:20.143318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c0bff-7d72-4e48-bf3e-e5057e7b9e95 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.143610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c0bff-7d72-4e48-bf3e-e5057e7b9e95 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.143947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c0bff-7d72-4e48-bf3e-e5057e7b9e95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.165729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f25c0bff-7d72-4e48-bf3e-e5057e7b9e95 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:20.166967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f25c0bff-7d72-4e48-bf3e-e5057e7b9e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:20.175270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 921/1845] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:20.185133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7037dbe-f39c-4da3-8c5e-c8341235231b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:20.191112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7037dbe-f39c-4da3-8c5e-c8341235231b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:20.195811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:20.196421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.196699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.196969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.197649 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:07:20.197951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:20.199547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:20.225642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 925/1846] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:07:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:07:20.225745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25c0bff-7d72-4e48-bf3e-e5057e7b9e95 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:20.230108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25c0bff-7d72-4e48-bf3e-e5057e7b9e95 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:20.236005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:20.236623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.236905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.237225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.237892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:20.239975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:20.329652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 922/1847] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:07:20 2022] POST /identity/v3/projects => generated 379 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:07:20.334948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:20.348183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620074a2-5c97-48f5-9450-23f7e3dcfa49 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:20.415076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:20.415730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.416011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.416289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.416927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:20.418925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:20.482240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 926/1848] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:07:20 2022] POST /identity/v3/users => generated 290 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:07:20.486673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:20.497362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d36dc1d-7178-4494-8592-e3bc941217c1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:20.502839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:20.503446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.503730 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.504006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.504646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:20.506278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:20.516741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 923/1849] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:20.520969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:20.531359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857010e-e4f2-48cc-98f4-7cd282b86a0d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:20.535945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617a9856-46a4-4a02-9a4e-7b1483097057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:20.536812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617a9856-46a4-4a02-9a4e-7b1483097057 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.541772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617a9856-46a4-4a02-9a4e-7b1483097057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.541772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617a9856-46a4-4a02-9a4e-7b1483097057 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.541772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617a9856-46a4-4a02-9a4e-7b1483097057 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:20.541772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617a9856-46a4-4a02-9a4e-7b1483097057 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:20.553946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 927/1850] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:20.558225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:20.568466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dffd830-560e-4af8-b5da-ad269b5f2d19 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:20.573051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:20.573744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.574026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.574302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] PATH_INFO: `/v3/projects/fac1485ccdeb434ca49caa52ebfdc24b/users/a5c784f29d2e4cb4bb8bd6d62fbc2216/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.589726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fac1485ccdeb434ca49caa52ebfdc24b, user_id=a5c784f29d2e4cb4bb8bd6d62fbc2216, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:20.593124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:20.664850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 924/1851] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:07:20 2022] PUT /identity/v3/projects/fac1485ccdeb434ca49caa52ebfdc24b/users/a5c784f29d2e4cb4bb8bd6d62fbc2216/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:07:20.670672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9355457-d429-40ce-aa95-8bee7a60c8fc None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.673498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9355457-d429-40ce-aa95-8bee7a60c8fc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.673498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9355457-d429-40ce-aa95-8bee7a60c8fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.692954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d9355457-d429-40ce-aa95-8bee7a60c8fc None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:20.694226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d9355457-d429-40ce-aa95-8bee7a60c8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:20.757283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 928/1852] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:20.767199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:20.768954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:20.778807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:20.828287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aaa1e3-493c-4f79-9b75-8f182a78d062 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:20.832590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:20.833684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:20.833802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:20.834057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:20.844364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:20.845839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:20.855199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 925/1853] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:07:26.546491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781dd778-a12c-429f-b93e-4937b8b95c1f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:26.547127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781dd778-a12c-429f-b93e-4937b8b95c1f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:26.547127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781dd778-a12c-429f-b93e-4937b8b95c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:26.597489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-781dd778-a12c-429f-b93e-4937b8b95c1f None None] MFA Rules not processed for user `a5c784f29d2e4cb4bb8bd6d62fbc2216`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:26.599237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-781dd778-a12c-429f-b93e-4937b8b95c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:26.661968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:26.703425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 929/1854] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:26 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:26.707928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:26.718233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e1378-9650-47ef-8e9d-58a6af206319 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:26.723275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:26.723889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:26.724205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:26.724480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:26.725126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:26.728374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:26.835927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 926/1855] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:07:26 2022] POST /identity/v3/projects => generated 379 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:07:26.839715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781dd778-a12c-429f-b93e-4937b8b95c1f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:26.851777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781dd778-a12c-429f-b93e-4937b8b95c1f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:26.924284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:26.924949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:26.925239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:26.925560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:26.926218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:26.928180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:27.026424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 930/1856] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:07:26 2022] POST /identity/v3/users => generated 296 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:07:27.027735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:27.043123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c4e7b0-9ef6-4648-9f58-d092c206eb33 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:27.049117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:27.049838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:27.050128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:27.050411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:27.051072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:27.052703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:27.071403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 927/1857] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:27.074519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:27.086663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94811a60-5040-4ffa-92fc-8ccf9072b172 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:27.092117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:27.092797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:27.093082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:27.093450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] PATH_INFO: `/v3/projects/9ab6cbbf0c2c43388ed1a33f737a2951/users/a113313fcf584b409de2da73424c1386/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:27.111860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ab6cbbf0c2c43388ed1a33f737a2951, user_id=a113313fcf584b409de2da73424c1386, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:27.117378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:27.209717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:27.211829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 931/1858] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:07:27 2022] PUT /identity/v3/projects/9ab6cbbf0c2c43388ed1a33f737a2951/users/a113313fcf584b409de2da73424c1386/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:27.221367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f3d29-216b-4703-a6fe-74b26ff1f195 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:27.315834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:27.316474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:27.316763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:27.317045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:27.317953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:27.319623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:27.333359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 928/1859] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:07:27.336042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:27.348569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37573ddc-b399-4aed-9387-fd6ecbfee867 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:27.353971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45309f6-5c30-48f5-a0ae-f018aa9450c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:27.357969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45309f6-5c30-48f5-a0ae-f018aa9450c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:27.357969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45309f6-5c30-48f5-a0ae-f018aa9450c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:27.357969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45309f6-5c30-48f5-a0ae-f018aa9450c1 admin admin] PATH_INFO: `/v3/projects/9ab6cbbf0c2c43388ed1a33f737a2951/users/a113313fcf584b409de2da73424c1386/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:27.358201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45309f6-5c30-48f5-a0ae-f018aa9450c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ab6cbbf0c2c43388ed1a33f737a2951, user_id=a113313fcf584b409de2da73424c1386, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:27.361004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45309f6-5c30-48f5-a0ae-f018aa9450c1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:27.385458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 932/1860] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:07:27 2022] PUT /identity/v3/projects/9ab6cbbf0c2c43388ed1a33f737a2951/users/a113313fcf584b409de2da73424c1386/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:29.770518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:29.773259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:29.796301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb685d9a-4527-4c08-9364-c1101c8d5e73 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:29.903104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:29.909618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:29.909618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:29.909618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] PATH_INFO: `/v3/users/6f9801053b5d45269d940c757f24b65d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:29.909618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f9801053b5d45269d940c757f24b65d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:29.909618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:29.930627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:30.148615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f9801053b5d45269d940c757f24b65d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:30.163468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f9801053b5d45269d940c757f24b65d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:30.260512 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f9801053b5d45269d940c757f24b65d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:30.261291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f9801053b5d45269d940c757f24b65d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:30.262251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f9801053b5d45269d940c757f24b65d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:30.281372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 929/1861] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:29 2022] DELETE /identity/v3/users/6f9801053b5d45269d940c757f24b65d => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:30.283779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:30.284229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:30.284329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:30.320165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:30.321959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:30.415738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 933/1862] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:07:30.426673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:30.428323 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:30.541668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bfc0d-b6b2-4e2e-80c5-266a1013a0c8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:30.545864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934f370f-4437-4885-969d-490078da241c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:30.546495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f370f-4437-4885-969d-490078da241c service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:30.546777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f370f-4437-4885-969d-490078da241c service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:30.547051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f370f-4437-4885-969d-490078da241c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:30.556948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934f370f-4437-4885-969d-490078da241c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:30.558469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934f370f-4437-4885-969d-490078da241c service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:30.571440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 930/1863] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:07:31.143747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:31.145284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:31.166300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919a4724-2081-4e93-8154-a120fec7af6e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:31.180920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:31.181741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:31.182129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:31.182586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] PATH_INFO: `/v3/projects/fa88a1f80f5f4af699682e7d4c4a3197` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:31.184005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa88a1f80f5f4af699682e7d4c4a3197)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:31.187327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:31.298552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:31.331701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa88a1f80f5f4af699682e7d4c4a3197', 'request_id': 'req-8805bac3-f10f-4f3a-914a-6e48f79732ac'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:31.340085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa88a1f80f5f4af699682e7d4c4a3197', 'request_id': 'req-8805bac3-f10f-4f3a-914a-6e48f79732ac'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:31.360255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa88a1f80f5f4af699682e7d4c4a3197', 'request_id': 'req-8805bac3-f10f-4f3a-914a-6e48f79732ac'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:31.477695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] The token cache is being invalidate because project fa88a1f80f5f4af699682e7d4c4a3197 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:07:31.477695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:31.477695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:07:31.492616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 934/1864] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:07:31 2022] DELETE /identity/v3/projects/fa88a1f80f5f4af699682e7d4c4a3197 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:07:34.248129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feefd560-88aa-453a-8d1c-2e70c9de309c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:34.248649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feefd560-88aa-453a-8d1c-2e70c9de309c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:34.248959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feefd560-88aa-453a-8d1c-2e70c9de309c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:34.279414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-feefd560-88aa-453a-8d1c-2e70c9de309c None None] MFA Rules not processed for user `a113313fcf584b409de2da73424c1386`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:34.280830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-feefd560-88aa-453a-8d1c-2e70c9de309c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:34.362776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 931/1865] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:34 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:34.376436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:34.377922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:34.456538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8805bac3-f10f-4f3a-914a-6e48f79732ac admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:07:34.463161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:07:34.463788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:34.464080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:34.464357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:34.466152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:34.531032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:07:34.532793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:07:34.544290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 935/1866] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:07:34 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:07:44.359892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:44.538858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=6,keystone:SELECT=70,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:07:53.380943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a83e3e-a3a0-4147-81bb-99d9ba41d940 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:07:53.381507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a83e3e-a3a0-4147-81bb-99d9ba41d940 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:07:53.381610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a83e3e-a3a0-4147-81bb-99d9ba41d940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:07:53.400882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-80a83e3e-a3a0-4147-81bb-99d9ba41d940 None None] MFA Rules not processed for user `724baaa4aa834e609fe4f513f9fc0b4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:07:53.402139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-80a83e3e-a3a0-4147-81bb-99d9ba41d940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:07:53.457783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 932/1867] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:07:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:07:59.939669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:07:59.941147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:00.018003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690bcd1-4be6-48f0-a1b4-1f95d8b52f63 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:00.025473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:00.026541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:00.026952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:00.027665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] PATH_INFO: `/v3/users/9fb0a0cc698f4e0eac1d2e9cb2da8a04` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:00.029147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fb0a0cc698f4e0eac1d2e9cb2da8a04)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:00.033609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:00.185305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fb0a0cc698f4e0eac1d2e9cb2da8a04'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:00.186325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fb0a0cc698f4e0eac1d2e9cb2da8a04'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:00.220999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fb0a0cc698f4e0eac1d2e9cb2da8a04'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:00.229358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fb0a0cc698f4e0eac1d2e9cb2da8a04'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:00.230657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fb0a0cc698f4e0eac1d2e9cb2da8a04'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:00.326677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 936/1868] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:07:59 2022] DELETE /identity/v3/users/9fb0a0cc698f4e0eac1d2e9cb2da8a04 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:08:00.329397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:00.330261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:00.330913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:00.352815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:00.354072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:00.416826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 933/1869] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:08:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:08:00.427022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:00.428504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:00.488389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a88a7c-5876-4dfa-adc3-7eba9cdc5329 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:00.492117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:00.492710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:00.492990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:00.493265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:00.502925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:00.504422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:00.513946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 937/1870] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:08:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:08:00.993201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:00.994679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:01.017701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78679134-2a62-42eb-8ad2-0e652c1a2f89 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:01.023906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:01.025051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:01.025607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:01.026461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] PATH_INFO: `/v3/projects/4ac3abcb19b745439c2fa02dc566fef2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:01.030254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ac3abcb19b745439c2fa02dc566fef2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:01.033035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:01.150991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ac3abcb19b745439c2fa02dc566fef2', 'request_id': 'req-e393965d-8a2c-442b-bb57-c707404c2c80'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.224922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ac3abcb19b745439c2fa02dc566fef2', 'request_id': 'req-e393965d-8a2c-442b-bb57-c707404c2c80'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.232113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ac3abcb19b745439c2fa02dc566fef2', 'request_id': 'req-e393965d-8a2c-442b-bb57-c707404c2c80'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.244265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] The token cache is being invalidate because project 4ac3abcb19b745439c2fa02dc566fef2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:08:01.244493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.245148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e393965d-8a2c-442b-bb57-c707404c2c80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.246783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 934/1871] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:08:00 2022] DELETE /identity/v3/projects/4ac3abcb19b745439c2fa02dc566fef2 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:08:01.251540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:01.261382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7c57a6-4dae-4c7b-b023-7843c4f2afab service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:01.289514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:01.354300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:01.355062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:01.355385 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:01.357178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] PATH_INFO: `/v3/users/9d110106608847608b03a0342af28aeb` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:01.357178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d110106608847608b03a0342af28aeb)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:01.358793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:01.542663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d110106608847608b03a0342af28aeb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.543569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d110106608847608b03a0342af28aeb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.559512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d110106608847608b03a0342af28aeb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.568509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d110106608847608b03a0342af28aeb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.569262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d110106608847608b03a0342af28aeb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:01.649192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 938/1872] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:08:01 2022] DELETE /identity/v3/users/9d110106608847608b03a0342af28aeb => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:08:01.732413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004638a4-6d5a-4d5c-bf1b-11f77ba8dfd0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:01.732955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004638a4-6d5a-4d5c-bf1b-11f77ba8dfd0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:01.732955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004638a4-6d5a-4d5c-bf1b-11f77ba8dfd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:01.754300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-004638a4-6d5a-4d5c-bf1b-11f77ba8dfd0 None None] MFA Rules not processed for user `ce7f52b7f96c400fb7d5fb0453c94362`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:01.756202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-004638a4-6d5a-4d5c-bf1b-11f77ba8dfd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:01.779675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:01.811796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 935/1873] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:08:01 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:08:02.142900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:02.144301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:02.245854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595c81de-10ae-4c5f-a532-701bfe714b1f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:02.253087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:02.253781 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:02.254257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:02.254538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] PATH_INFO: `/v3/projects/07d5ce8a62dc4c33a890f1cdc6cc9856` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:02.255701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07d5ce8a62dc4c33a890f1cdc6cc9856)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:02.257641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:02.392657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:02.400635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07d5ce8a62dc4c33a890f1cdc6cc9856', 'request_id': 'req-1eb703bb-8cef-420d-b152-068a50258e32'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:02.409604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07d5ce8a62dc4c33a890f1cdc6cc9856', 'request_id': 'req-1eb703bb-8cef-420d-b152-068a50258e32'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:02.429505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07d5ce8a62dc4c33a890f1cdc6cc9856', 'request_id': 'req-1eb703bb-8cef-420d-b152-068a50258e32'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:02.527599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] The token cache is being invalidate because project 07d5ce8a62dc4c33a890f1cdc6cc9856 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:08:02.527895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:02.528559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1eb703bb-8cef-420d-b152-068a50258e32 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:08:02.530138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 939/1874] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:08:02 2022] DELETE /identity/v3/projects/07d5ce8a62dc4c33a890f1cdc6cc9856 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:08:02.535589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d00bbb-f8cd-4f79-9071-87c9a0e90e30 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:02.535904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d00bbb-f8cd-4f79-9071-87c9a0e90e30 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:02.536181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d00bbb-f8cd-4f79-9071-87c9a0e90e30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:02.562182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-53d00bbb-f8cd-4f79-9071-87c9a0e90e30 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:02.564685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-53d00bbb-f8cd-4f79-9071-87c9a0e90e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:02.700157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcc47b3-bd04-4088-bf7e-88fed24e1a22 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:02.700467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcc47b3-bd04-4088-bf7e-88fed24e1a22 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:02.700723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcc47b3-bd04-4088-bf7e-88fed24e1a22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:02.707793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 936/1875] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:08:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:08:02.727041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-9dcc47b3-bd04-4088-bf7e-88fed24e1a22 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:02.742970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9dcc47b3-bd04-4088-bf7e-88fed24e1a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:02.754085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 940/1876] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:08:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:08:02.758156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d00bbb-f8cd-4f79-9071-87c9a0e90e30 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:02.769849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d00bbb-f8cd-4f79-9071-87c9a0e90e30 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:02.777957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:02.778919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:02.779403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:02.779853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:02.781050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:08:02.781874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:02.784024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:02.817456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 937/1877] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:08:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:08:02.823351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcc47b3-bd04-4088-bf7e-88fed24e1a22 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:02.834217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcc47b3-bd04-4088-bf7e-88fed24e1a22 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:02.839590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:02.840218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:02.840500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:02.840780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:02.841793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:02.843829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:02.968211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 941/1878] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:08:02 2022] POST /identity/v3/projects => generated 381 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:08:02.973651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:02.986139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b95e8e-5b28-4349-b098-60fc746ef20a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:03.064537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:03.065173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:03.065493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:03.065775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:03.066414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:03.068341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:03.194948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 938/1879] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:08:02 2022] POST /identity/v3/users => generated 291 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:08:03.199345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:03.216033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18c87a3-8f2d-493c-968b-ed49d3e1ccb4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:03.221658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:03.222289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:03.222577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:03.222857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:03.223586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:03.225212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:03.236975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 942/1880] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:08:03 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:08:03.241049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:03.253135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bc7b6b-92e3-4a98-9740-532a98e53f77 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:03.258695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3139d1d8-39ce-4697-97ab-08b36a7bb21e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:03.259320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3139d1d8-39ce-4697-97ab-08b36a7bb21e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:03.259611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3139d1d8-39ce-4697-97ab-08b36a7bb21e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:03.259891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3139d1d8-39ce-4697-97ab-08b36a7bb21e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:03.260539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3139d1d8-39ce-4697-97ab-08b36a7bb21e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:03.262193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3139d1d8-39ce-4697-97ab-08b36a7bb21e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:03.278213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 939/1881] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:08:03 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:08:03.281545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:03.296023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6544bc1-cf1b-463e-8a1b-26a208937645 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:03.314091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:03.314091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:03.314091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:03.314091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] PATH_INFO: `/v3/projects/d709b2d4019d485eb6dfa96e1b3ec9ee/users/6cff10219cb84ae7945e821b2cc65a11/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:03.322396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d709b2d4019d485eb6dfa96e1b3ec9ee, user_id=6cff10219cb84ae7945e821b2cc65a11, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:03.325260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:03.448444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 943/1882] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:08:03 2022] PUT /identity/v3/projects/d709b2d4019d485eb6dfa96e1b3ec9ee/users/6cff10219cb84ae7945e821b2cc65a11/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:08:03.455613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a742cd-5641-405b-b71c-0a2bb8cda6dd None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:03.455906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a742cd-5641-405b-b71c-0a2bb8cda6dd None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:03.456184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a742cd-5641-405b-b71c-0a2bb8cda6dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:03.490558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-25a742cd-5641-405b-b71c-0a2bb8cda6dd None None] MFA Rules not processed for user `6cff10219cb84ae7945e821b2cc65a11`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:03.491374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-25a742cd-5641-405b-b71c-0a2bb8cda6dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:03.526301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:03.579688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 940/1883] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:08:03 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:08:03.584312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:03.679475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe5a26c-edbd-47d0-86b2-756e2c422fd8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:03.686488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:03.687128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:03.687421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:03.688254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:03.688906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:03.690928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:03.805201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 944/1884] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:08:03 2022] POST /identity/v3/projects => generated 381 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:08:03.809577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a742cd-5641-405b-b71c-0a2bb8cda6dd None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:03.823852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a742cd-5641-405b-b71c-0a2bb8cda6dd None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:03.907351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:03.907911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:03.908200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:03.908480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:03.909127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:03.911394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:04.032518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 941/1885] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:08:03 2022] POST /identity/v3/users => generated 297 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:08:04.037012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:04.047688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376e2420-44b6-4587-b194-b63e5bd91126 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:04.053436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6031c2-90c0-4dfe-9318-ea9c74814c91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:04.054398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6031c2-90c0-4dfe-9318-ea9c74814c91 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.054749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6031c2-90c0-4dfe-9318-ea9c74814c91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.055095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6031c2-90c0-4dfe-9318-ea9c74814c91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.055847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6031c2-90c0-4dfe-9318-ea9c74814c91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:04.057591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6031c2-90c0-4dfe-9318-ea9c74814c91 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:04.069991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 945/1886] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:08:04 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:08:04.074500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:04.086747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb0678a-c3c6-4cc7-9c15-244fbfb2f36c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:04.091938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:04.092863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.093154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.093481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] PATH_INFO: `/v3/projects/557e582073d24cf38db4fcf40d6b698b/users/e641396646164820b4c028f3bc9b4285/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.112113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=557e582073d24cf38db4fcf40d6b698b, user_id=e641396646164820b4c028f3bc9b4285, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:04.115315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:04.134042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e56ffd-758d-4b48-a965-4caec333fe49 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.134327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e56ffd-758d-4b48-a965-4caec333fe49 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.134605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e56ffd-758d-4b48-a965-4caec333fe49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.154851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-77e56ffd-758d-4b48-a965-4caec333fe49 None None] MFA Rules not processed for user `a5c784f29d2e4cb4bb8bd6d62fbc2216`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:04.156057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-77e56ffd-758d-4b48-a965-4caec333fe49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:04.185947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 942/1887] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:08:04 2022] PUT /identity/v3/projects/557e582073d24cf38db4fcf40d6b698b/users/e641396646164820b4c028f3bc9b4285/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:08:04.190582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:04.197894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:04.215834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 946/1888] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:08:04 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:08:04.257779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555561c9-5304-47b9-a516-ed1736338d16 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:04.262425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90d694-a48d-43ca-b826-6ebc4d0d0e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:04.263037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90d694-a48d-43ca-b826-6ebc4d0d0e24 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.263349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90d694-a48d-43ca-b826-6ebc4d0d0e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.263631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90d694-a48d-43ca-b826-6ebc4d0d0e24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.264293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90d694-a48d-43ca-b826-6ebc4d0d0e24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:04.265988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90d694-a48d-43ca-b826-6ebc4d0d0e24 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:04.276740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 943/1889] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:08:04 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:08:04.280953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e56ffd-758d-4b48-a965-4caec333fe49 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:04.292447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e56ffd-758d-4b48-a965-4caec333fe49 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:04.298105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:04.298863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.299155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.299437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] PATH_INFO: `/v3/projects/557e582073d24cf38db4fcf40d6b698b/users/e641396646164820b4c028f3bc9b4285/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.301414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] RBAC: Authorizing `identity:create_grant(project_id=557e582073d24cf38db4fcf40d6b698b, user_id=e641396646164820b4c028f3bc9b4285, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:04.304192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:04.324779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 947/1890] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:08:04 2022] PUT /identity/v3/projects/557e582073d24cf38db4fcf40d6b698b/users/e641396646164820b4c028f3bc9b4285/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:08:04.326696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4f93ea-43b2-4258-a1dc-7889fc999e5e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.326930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4f93ea-43b2-4258-a1dc-7889fc999e5e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.327210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4f93ea-43b2-4258-a1dc-7889fc999e5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.359911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0a4f93ea-43b2-4258-a1dc-7889fc999e5e None None] MFA Rules not processed for user `e641396646164820b4c028f3bc9b4285`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:04.361117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0a4f93ea-43b2-4258-a1dc-7889fc999e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:04.380448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:04.431773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 944/1891] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:08:04 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:08:04.444692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:04.446504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:04.506689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34877c7a-ccc4-455f-b53f-f4d974f10282 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:04.511233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:04.511924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.512339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.512737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.524242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:04.525844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:04.536753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 948/1892] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:08:04 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:08:04.727202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4f93ea-43b2-4258-a1dc-7889fc999e5e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:04.737702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4f93ea-43b2-4258-a1dc-7889fc999e5e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:04.742109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5397bcb0-c0e9-43b7-bc48-d725b127febc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:04.742934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5397bcb0-c0e9-43b7-bc48-d725b127febc service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:04.743282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5397bcb0-c0e9-43b7-bc48-d725b127febc service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:04.743880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5397bcb0-c0e9-43b7-bc48-d725b127febc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:04.834296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5397bcb0-c0e9-43b7-bc48-d725b127febc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:04.836101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5397bcb0-c0e9-43b7-bc48-d725b127febc service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:04.845986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 945/1893] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:08:04 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:08:14.531375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=37,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:14.841865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=48 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:08:52.184651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:52.186846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:52.299732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ea422-46dd-44a3-9fc9-8fbc22f6c65c service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:52.305365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:52.306223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:52.306668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:52.307290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:52.309087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:52.321915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:52.323504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:52.334976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 949/1894] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:08:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:08:55.878781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ba082-a899-47d9-baa8-14f6a2b6192a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:55.879278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ba082-a899-47d9-baa8-14f6a2b6192a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:55.879391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ba082-a899-47d9-baa8-14f6a2b6192a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:55.902635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d94ba082-a899-47d9-baa8-14f6a2b6192a None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:08:55.902635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d94ba082-a899-47d9-baa8-14f6a2b6192a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:55.957469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 946/1895] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:08:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:08:56.000438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:08:56.002707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:08:56.088572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8de9dc-6225-4917-9dc8-c10787774275 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:08:56.093686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10f11ab-084e-400c-ac34-8da4f9843d50 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:08:56.094303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10f11ab-084e-400c-ac34-8da4f9843d50 service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:08:56.095132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10f11ab-084e-400c-ac34-8da4f9843d50 service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:08:56.095556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10f11ab-084e-400c-ac34-8da4f9843d50 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:08:56.104801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10f11ab-084e-400c-ac34-8da4f9843d50 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:08:56.106586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10f11ab-084e-400c-ac34-8da4f9843d50 service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:08:56.119391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 950/1896] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:08:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:09:05.950929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=25 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:09:06.114146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=64 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:09:17.570434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d82e0b4-292c-441f-87d2-38294eed8e2c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:17.570434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d82e0b4-292c-441f-87d2-38294eed8e2c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:17.570434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d82e0b4-292c-441f-87d2-38294eed8e2c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:17.570434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 947/1897] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:09:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:09:17.574722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35adb166-5c12-4ec6-81dc-08879bf895fa None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:17.575151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35adb166-5c12-4ec6-81dc-08879bf895fa None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:17.575271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35adb166-5c12-4ec6-81dc-08879bf895fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:17.606033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-35adb166-5c12-4ec6-81dc-08879bf895fa None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:09:17.606033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-35adb166-5c12-4ec6-81dc-08879bf895fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:09:17.684311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 951/1898] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:09:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:09:17.696980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d82e0b4-292c-441f-87d2-38294eed8e2c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:09:17.698678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0d82e0b4-292c-441f-87d2-38294eed8e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:09:17.780260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d82e0b4-292c-441f-87d2-38294eed8e2c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:09:17.784481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574c2764-720e-4329-b8e6-16975ceda80c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7fae443ad92439aadd2f94ad3bca594', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:09:17.785093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574c2764-720e-4329-b8e6-16975ceda80c service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:17.785408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574c2764-720e-4329-b8e6-16975ceda80c service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:17.785687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574c2764-720e-4329-b8e6-16975ceda80c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:17.796006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574c2764-720e-4329-b8e6-16975ceda80c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:09:17.798215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574c2764-720e-4329-b8e6-16975ceda80c service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:09:17.808848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 948/1899] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:09:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:09:18.033467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec8ac9-84f6-4ded-a31b-884c3b402bd8 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:18.033893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec8ac9-84f6-4ded-a31b-884c3b402bd8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:18.034207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec8ac9-84f6-4ded-a31b-884c3b402bd8 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:18.035591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 952/1900] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:09:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:09:18.040831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d627d7-d853-4a92-8fe4-0cd877e3018c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:18.043031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d627d7-d853-4a92-8fe4-0cd877e3018c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:18.043031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d627d7-d853-4a92-8fe4-0cd877e3018c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:18.061158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-89d627d7-d853-4a92-8fe4-0cd877e3018c None None] MFA Rules not processed for user `91639dc527ae47e892164544572cbec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:09:18.062431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-89d627d7-d853-4a92-8fe4-0cd877e3018c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:09:18.113171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 949/1901] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:09:18 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:09:18.125026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ec8ac9-84f6-4ded-a31b-884c3b402bd8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:09:18.126900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f6ec8ac9-84f6-4ded-a31b-884c3b402bd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:09:18.139818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ec8ac9-84f6-4ded-a31b-884c3b402bd8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:09:18.144000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ee946-a5af-4c9b-8567-a1c9aba091e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:09:18.144625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ee946-a5af-4c9b-8567-a1c9aba091e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:18.144904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ee946-a5af-4c9b-8567-a1c9aba091e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:18.145176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ee946-a5af-4c9b-8567-a1c9aba091e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:18.152063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458ee946-a5af-4c9b-8567-a1c9aba091e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:09:18.153509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458ee946-a5af-4c9b-8567-a1c9aba091e3 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:09:18.161231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 953/1902] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:09:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:09:18.245167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb55ce3d-9aa3-4572-9f10-0fa44949a4ed None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:18.245507 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb55ce3d-9aa3-4572-9f10-0fa44949a4ed None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:18.245816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb55ce3d-9aa3-4572-9f10-0fa44949a4ed None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:18.247181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 950/1903] 10.222.0.66 () {54 vars in 921 bytes} [Fri Oct 28 20:09:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:09:18.252495 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627db17c-4074-4be2-899c-52044e2304e3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:18.252783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627db17c-4074-4be2-899c-52044e2304e3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:18.253072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627db17c-4074-4be2-899c-52044e2304e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:18.273092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-627db17c-4074-4be2-899c-52044e2304e3 None None] MFA Rules not processed for user `d54e8c38f0f84e4287b0445947bde916`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:09:18.274349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-627db17c-4074-4be2-899c-52044e2304e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:09:18.284486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 954/1904] 10.222.0.66 () {58 vars in 1018 bytes} [Fri Oct 28 20:09:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:09:18.295188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb55ce3d-9aa3-4572-9f10-0fa44949a4ed None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:09:18.297489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cb55ce3d-9aa3-4572-9f10-0fa44949a4ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:09:18.310885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb55ce3d-9aa3-4572-9f10-0fa44949a4ed None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:09:18.315167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:09:18.315633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:09:18.315919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:09:18.316199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:09:18.326149 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:09:18.327573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:09:18.339173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 951/1905] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:09:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:09:28.267806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=40 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:09:28.334298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=64 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:10:22.427652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627db17c-4074-4be2-899c-52044e2304e3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:10:22.438556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627db17c-4074-4be2-899c-52044e2304e3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:10:22.442397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:10:22.443012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:10:22.443296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:10:22.443573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:10:22.445428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:22.461506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:10:22.462926 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:10:22.475536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 955/1906] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:10:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:10:24.848477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:10:24.857878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534e540-eef8-4bee-9779-3b77ee40c7a3 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:10:24.861508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:10:24.862113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:10:24.862395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:10:24.862692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:10:24.864466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:24.878361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:10:24.879779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:10:24.892960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 952/1907] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:10:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:10:32.470969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:10:33.713250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:10:33.714831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:33.728507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5911cc98-5caa-4903-b758-83849b501fd2 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:10:33.734699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:10:33.735299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:10:33.735577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:10:33.735860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:10:33.737682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:33.795329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:10:33.796979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:10:33.806279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 956/1908] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:10:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:10:34.888027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:10:38.505481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:10:38.514928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:38.527541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569ca37d-42da-4f0f-a103-8608515f8e68 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:10:38.532359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:10:38.532621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:10:38.532966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:10:38.533345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:10:38.535153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:38.603076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:10:38.604495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:10:38.614257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 953/1909] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:10:38 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:10:40.773152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:10:40.789222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09317f04-cb1b-48df-8e79-6fecce2912d4 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:10:40.794657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:10:40.795565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:10:40.795867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:10:40.796145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:10:40.798994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:40.863638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:10:40.865153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:10:40.874945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 957/1910] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:10:40 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:10:42.906424 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:10:42.915098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abf499-2a42-4281-a0cd-4e0c990ecc8e service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:10:42.919860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:10:42.920877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:10:42.921232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:10:42.921548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:10:42.923457 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:42.990592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:10:42.991972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:10:43.003258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 954/1911] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:10:42 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:10:50.869948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=56 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:10:52.997170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=56 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:10:59.733638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:10:59.745223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ffe460-bc37-4d8a-91b7-689d7fb62da2 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:10:59.751281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:10:59.752006 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:10:59.752358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:10:59.752706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:10:59.754611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:10:59.767972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:10:59.769693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:10:59.781306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 958/1912] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:10:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:11:09.776106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:11:36.786428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:11:36.796269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388b767-9f1b-4e03-b24d-2a1a74a8ef1a service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:11:36.799906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:11:36.800524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:11:36.800808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:11:36.801086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:11:36.802848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:11:36.813453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:11:36.814882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:11:36.823299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 955/1913] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:11:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:11:46.819997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:11:51.637087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:11:51.647203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e1d9cd-506d-4b0f-b7c9-01e25ad18d26 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:11:51.650797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:11:51.651399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:11:51.651686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:11:51.651963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:11:51.654219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:11:51.714678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:11:51.716607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:11:51.725433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 959/1914] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:11:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:12:01.721382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=28 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:12:33.674712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:12:33.676298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:12:33.693462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a26291-c50d-48c3-b397-62e9bdcd41f9 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:12:33.699753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31bbf51-3001-4d69-ad3a-3d3f720644a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:12:33.700371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31bbf51-3001-4d69-ad3a-3d3f720644a9 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:12:33.700652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31bbf51-3001-4d69-ad3a-3d3f720644a9 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:12:33.700926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31bbf51-3001-4d69-ad3a-3d3f720644a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:12:33.702666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e31bbf51-3001-4d69-ad3a-3d3f720644a9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:12:33.760623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31bbf51-3001-4d69-ad3a-3d3f720644a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:12:33.762237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31bbf51-3001-4d69-ad3a-3d3f720644a9 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:12:33.771058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 956/1915] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:12:33 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:12:43.767045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=28 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:13:04.726688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:13:04.736058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c264ebe9-2dbc-49a3-a64d-459a5b53dc04 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:13:04.739720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8cbd04-3eb5-48a3-bd23-b9090dfe5d18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:13:04.740357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8cbd04-3eb5-48a3-bd23-b9090dfe5d18 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:13:04.740635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8cbd04-3eb5-48a3-bd23-b9090dfe5d18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:13:04.740905 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8cbd04-3eb5-48a3-bd23-b9090dfe5d18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:13:04.750410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8cbd04-3eb5-48a3-bd23-b9090dfe5d18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:13:04.751846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8cbd04-3eb5-48a3-bd23-b9090dfe5d18 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:13:04.760751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 960/1916] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:13:04 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:13:14.756618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:14:06.016883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f412b-38c6-4830-a0f1-67efb69eebd8 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:14:06.017460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f412b-38c6-4830-a0f1-67efb69eebd8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:14:06.017558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f412b-38c6-4830-a0f1-67efb69eebd8 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:14:06.018865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 957/1917] 10.222.0.66 () {54 vars in 912 bytes} [Fri Oct 28 20:14:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:14:06.024964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:14:06.025464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:14:06.025578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:14:06.107268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:14:06.108489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:14:06.180534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 961/1918] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:14:06 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:14:06.203852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f412b-38c6-4830-a0f1-67efb69eebd8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:14:06.218054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f412b-38c6-4830-a0f1-67efb69eebd8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:14:06.229743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4cc595-79f7-42db-8d99-a133aa7a40a5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:14:06.230363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4cc595-79f7-42db-8d99-a133aa7a40a5 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:14:06.230647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4cc595-79f7-42db-8d99-a133aa7a40a5 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:14:06.230928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4cc595-79f7-42db-8d99-a133aa7a40a5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:14:06.240281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4cc595-79f7-42db-8d99-a133aa7a40a5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:14:06.241905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4cc595-79f7-42db-8d99-a133aa7a40a5 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:14:06.251692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 958/1919] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:14:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:14:16.178916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=48 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:14:16.247517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:14:37.844963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:14:37.846387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:14:37.860084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca89873a-162a-4d68-8998-e119972c4fa5 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:14:37.865167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:14:37.865909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:14:37.866351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:14:37.866640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:14:37.875020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:14:37.876604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:14:37.886421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 962/1920] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:14:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:14:47.882207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:15:17.907789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dd51f-ee98-44a9-a1c9-b5abd9818d9e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:17.908297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dd51f-ee98-44a9-a1c9-b5abd9818d9e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:17.908679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dd51f-ee98-44a9-a1c9-b5abd9818d9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:17.928714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-af5dd51f-ee98-44a9-a1c9-b5abd9818d9e None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:15:17.930037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-af5dd51f-ee98-44a9-a1c9-b5abd9818d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:15:17.952388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 959/1921] 10.222.0.66 () {58 vars in 967 bytes} [Fri Oct 28 20:15:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:15:18.119477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:18.131332 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c2b84-4691-466c-922a-ef390a5c3d48 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:18.136718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab77e48e-edbe-4e07-8248-e0d626fa0fda service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:18.137339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab77e48e-edbe-4e07-8248-e0d626fa0fda service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:18.137641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab77e48e-edbe-4e07-8248-e0d626fa0fda service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:18.137925 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab77e48e-edbe-4e07-8248-e0d626fa0fda service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:18.146883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab77e48e-edbe-4e07-8248-e0d626fa0fda service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:18.148533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab77e48e-edbe-4e07-8248-e0d626fa0fda service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:18.157744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 963/1922] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:15:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:15:19.978301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dd51f-ee98-44a9-a1c9-b5abd9818d9e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:20.003810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dd51f-ee98-44a9-a1c9-b5abd9818d9e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:20.010883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0251d73-429a-4c69-96b2-75a88fce577e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:20.012087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0251d73-429a-4c69-96b2-75a88fce577e service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:20.012655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0251d73-429a-4c69-96b2-75a88fce577e service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:20.013147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0251d73-429a-4c69-96b2-75a88fce577e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:20.015540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e0251d73-429a-4c69-96b2-75a88fce577e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:15:20.060217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0251d73-429a-4c69-96b2-75a88fce577e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:20.062538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0251d73-429a-4c69-96b2-75a88fce577e service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:20.075983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 960/1923] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:20.209749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7389e-041d-42f4-8b3d-52a5a88cf869 None None] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:20.210781 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7389e-041d-42f4-8b3d-52a5a88cf869 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:20.211170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7389e-041d-42f4-8b3d-52a5a88cf869 None None] PATH_INFO: `/` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:20.213117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 964/1924] 10.222.0.66 () {54 vars in 912 bytes} [Fri Oct 28 20:15:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 20:15:20.220010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833d780e-8431-4e53-8858-5cd448a32382 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:20.220417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833d780e-8431-4e53-8858-5cd448a32382 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:20.220906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833d780e-8431-4e53-8858-5cd448a32382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:20.249654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-833d780e-8431-4e53-8858-5cd448a32382 None None] MFA Rules not processed for user `5fbacc77af9b43f1a2c1cf15cd09e337`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:15:20.249654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-833d780e-8431-4e53-8858-5cd448a32382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:15:20.259212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 961/1925] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:15:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:15:20.285156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad7389e-041d-42f4-8b3d-52a5a88cf869 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:20.309036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad7389e-041d-42f4-8b3d-52a5a88cf869 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:20.314341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:20.315112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:20.315465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:20.315796 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:20.324660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:20.326357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:20.337292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 965/1926] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:24.048094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e785e93-6a1f-40e9-871e-2fcd08693ce0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:24.048536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e785e93-6a1f-40e9-871e-2fcd08693ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:24.048659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e785e93-6a1f-40e9-871e-2fcd08693ce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:24.098002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0e785e93-6a1f-40e9-871e-2fcd08693ce0 None None] MFA Rules not processed for user `e0ba6bf2459d4a98a3a1f5684bdad2b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:15:24.099118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0e785e93-6a1f-40e9-871e-2fcd08693ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:15:24.107534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 962/1927] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:15:24 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:15:24.424221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:24.448245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7bf4a4-deb1-4a33-8aee-1e2fba265e06 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:24.453633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:24.454252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:24.454531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:24.454979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:24.456940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:15:24.484255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:24.485740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:24.494751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 966/1928] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:24.889587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e785e93-6a1f-40e9-871e-2fcd08693ce0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:24.895141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:24.902304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e785e93-6a1f-40e9-871e-2fcd08693ce0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:24.904112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025a931d-e100-4739-9eda-4a4d16bd241d service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:24.907268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:24.907603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:24.907880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:24.908167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:24.908274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:24.908444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:24.908557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:24.908719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:24.910191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:15:24.920982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:24.923572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:24.925673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:24.928772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:24.936045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 967/1929] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:15:24.941168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 963/1930] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:29.463399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:29.473380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a111c2-942d-43c8-8d50-ada988a6c389 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:29.477948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:29.478581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:29.478868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:29.479143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:29.492033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:29.493524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:29.506499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 968/1931] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:33.987197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:34.000155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcb941d-3b85-4353-8ab2-70382383a8db service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:34.006199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:34.006855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:34.007135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:34.007405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:34.015495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:34.017223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:34.045699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 964/1932] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:34.632467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:34.643620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b64f7dd-f7e1-445e-a453-e25fa1846c6f service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:34.649896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:34.651617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:34.651617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:34.651617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:34.652951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:15:34.665983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:34.667596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:34.678869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 969/1933] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:39.550020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:39.570206 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d152ee-aa35-40da-8896-fb64773fb8d4 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:39.573791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:39.574401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:39.574680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:39.574960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:39.584969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:39.586442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:39.614191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 965/1934] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:39 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:42.834660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:15:42.843826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ad24d2-0101-46be-b825-9432569a7f04 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:15:42.847438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:15:42.848037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:15:42.848319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:15:42.848592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:15:42.857613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:15:42.859029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:15:42.885883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 970/1935] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:15:42 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:15:49.611531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=72 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:15:52.883227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=58 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:16:14.372943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:16:14.388016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96349ce8-48d5-43c2-8281-24e99140151a service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:16:14.397405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:16:14.398272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:16:14.398589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:16:14.398874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:16:14.408165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:16:14.409992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:16:14.431628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 966/1936] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:16:14 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:16:24.432045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=10 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:16:38.914842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:16:38.924941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb81422-036b-4329-a739-b1c589ab11bf service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:16:38.928465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:16:38.929131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:16:38.929351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:16:38.929699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:16:38.936972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:16:38.938357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:16:38.946091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 971/1937] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:16:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 32 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:16:48.942602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:16:51.694405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:16:51.704296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea504c3f-953e-49e2-8977-183e6edb0b66 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:16:51.707935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:16:51.708535 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:16:51.708820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:16:51.709095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:16:51.718797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:16:51.720232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:16:51.748278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 967/1938] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:16:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:17:01.745738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=13 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:17:33.592790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:17:33.605877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea042ad-9df6-4cf3-8f6e-15aa7bc71815 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:17:33.611963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:17:33.612592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:17:33.612872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:17:33.613147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:17:33.622517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:17:33.623940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:17:33.651311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 972/1939] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:17:33 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:17:43.648637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=13 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:18:04.749572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:18:04.751140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:18:04.819568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174fba2a-f48b-4791-910a-2b3c39e87285 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:18:04.823272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:18:04.823830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:18:04.824115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:18:04.824394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:18:04.826225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:18:04.906724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:18:04.908158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:18:04.926392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 968/1940] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:18:04 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:18:14.923859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=66 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:19:41.796829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:19:41.798429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:19:41.879725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339d7d02-585c-48f4-85bf-ee767a9d3e0b service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:19:41.893069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:19:41.893809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:19:41.894207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:19:41.894606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:19:41.896560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:19:41.909344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:19:41.910939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:19:41.921829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 973/1941] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:19:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:19:51.916774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=32 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:06.538134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:06.548445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dbfd17-69ae-4a91-a4a0-ef2a5a5d5bbd service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:06.552234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311b9fda-f7a7-4364-a587-eec02d3f9157 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:06.552958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b9fda-f7a7-4364-a587-eec02d3f9157 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:06.553346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b9fda-f7a7-4364-a587-eec02d3f9157 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:06.553632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b9fda-f7a7-4364-a587-eec02d3f9157 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:06.555571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-311b9fda-f7a7-4364-a587-eec02d3f9157 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:06.644837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311b9fda-f7a7-4364-a587-eec02d3f9157 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:06.646307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311b9fda-f7a7-4364-a587-eec02d3f9157 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:06.665613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 969/1942] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:06 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:06.913304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:06.914961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:07.009117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3865228d-602d-4fc7-9077-da571339be53 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:07.014491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:07.019986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:07.019986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:07.019986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] PATH_INFO: `/v3/users/6cff10219cb84ae7945e821b2cc65a11` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:07.019986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cff10219cb84ae7945e821b2cc65a11)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:07.019986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:07.203204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cff10219cb84ae7945e821b2cc65a11'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:07.203914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cff10219cb84ae7945e821b2cc65a11'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:07.216064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cff10219cb84ae7945e821b2cc65a11'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:07.228435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cff10219cb84ae7945e821b2cc65a11'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:07.229151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cff10219cb84ae7945e821b2cc65a11'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:07.308579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 974/1943] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:20:06 2022] DELETE /identity/v3/users/6cff10219cb84ae7945e821b2cc65a11 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:07.314130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:07.314428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:07.314711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:07.343966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:07.345180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:07.404561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 970/1944] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:07.414214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:07.415661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:07.475747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b94997a-909c-4512-b924-e6b29723952c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:07.479449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:07.480060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:07.480339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:07.480609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:07.487406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:07.491296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:07.492778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:07.503323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 975/1945] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:08.007266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:08.009250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:08.042161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a7c0d8-3ef4-45f7-aa84-c8c0406fc44c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:08.048690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:08.048983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:08.049494 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:08.049740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:08.050008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] PATH_INFO: `/v3/projects/d709b2d4019d485eb6dfa96e1b3ec9ee` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:08.050638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:08.051144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] RBAC: Authorizing `identity:delete_project(project_id=d709b2d4019d485eb6dfa96e1b3ec9ee)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:08.053050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:08.062910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c3cec0-7eec-4622-9dda-991bf8448393 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:08.066677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:08.067319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:08.067609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:08.067888 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:08.069669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:08.082330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:08.083767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:08.095910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 976/1946] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:20:08.171661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd709b2d4019d485eb6dfa96e1b3ec9ee', 'request_id': 'req-ec377e29-7152-4c37-af96-d446b3487ccf'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.175714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:08.265266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd709b2d4019d485eb6dfa96e1b3ec9ee', 'request_id': 'req-ec377e29-7152-4c37-af96-d446b3487ccf'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.272252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd709b2d4019d485eb6dfa96e1b3ec9ee', 'request_id': 'req-ec377e29-7152-4c37-af96-d446b3487ccf'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.283832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] The token cache is being invalidate because project d709b2d4019d485eb6dfa96e1b3ec9ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:20:08.284124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.284780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.289188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 971/1947] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:20:08 2022] DELETE /identity/v3/projects/d709b2d4019d485eb6dfa96e1b3ec9ee => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:08.291201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:08.302409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45876cb9-7899-4803-b251-5b5f244b81b6 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:08.364273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:08.364824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:08.365102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:08.365412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] PATH_INFO: `/v3/users/e641396646164820b4c028f3bc9b4285` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:08.366524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] RBAC: Authorizing `identity:delete_user(user_id=e641396646164820b4c028f3bc9b4285)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:08.368439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:08.517576 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e641396646164820b4c028f3bc9b4285'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.518351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e641396646164820b4c028f3bc9b4285'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.531525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e641396646164820b4c028f3bc9b4285'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.538641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e641396646164820b4c028f3bc9b4285'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.539353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1524a19d-131a-4efc-ad85-65dcf1fdbdad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e641396646164820b4c028f3bc9b4285'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:08.590884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 977/1948] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:20:08 2022] DELETE /identity/v3/users/e641396646164820b4c028f3bc9b4285 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:09.156936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:09.160180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:09.178884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec377e29-7152-4c37-af96-d446b3487ccf admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:09.265532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:09.266291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:09.267526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:09.267923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] PATH_INFO: `/v3/projects/557e582073d24cf38db4fcf40d6b698b` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:09.269755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=557e582073d24cf38db4fcf40d6b698b)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:09.273246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:09.389426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '557e582073d24cf38db4fcf40d6b698b', 'request_id': 'req-54d28742-90aa-49ef-93c7-75c5d43132a8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:09.446586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '557e582073d24cf38db4fcf40d6b698b', 'request_id': 'req-54d28742-90aa-49ef-93c7-75c5d43132a8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:09.454408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '557e582073d24cf38db4fcf40d6b698b', 'request_id': 'req-54d28742-90aa-49ef-93c7-75c5d43132a8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:09.469966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] The token cache is being invalidate because project 557e582073d24cf38db4fcf40d6b698b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:20:09.470292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:09.470946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-54d28742-90aa-49ef-93c7-75c5d43132a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:09.477159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 972/1949] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:20:09 2022] DELETE /identity/v3/projects/557e582073d24cf38db4fcf40d6b698b => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:09.478858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19d20ce-88c1-4055-8da7-1c1861f81a04 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:09.484396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19d20ce-88c1-4055-8da7-1c1861f81a04 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:09.484396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19d20ce-88c1-4055-8da7-1c1861f81a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:09.522698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a19d20ce-88c1-4055-8da7-1c1861f81a04 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:09.524068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a19d20ce-88c1-4055-8da7-1c1861f81a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:09.569533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:09.594546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 978/1950] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:20:09.603579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0718760e-4a66-48a9-ae75-d2e74d9493cf None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:09.603579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0718760e-4a66-48a9-ae75-d2e74d9493cf None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:09.603579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0718760e-4a66-48a9-ae75-d2e74d9493cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:09.627585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0718760e-4a66-48a9-ae75-d2e74d9493cf None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:09.628745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0718760e-4a66-48a9-ae75-d2e74d9493cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:09.640954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 973/1951] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:09.663417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19d20ce-88c1-4055-8da7-1c1861f81a04 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:09.668994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19d20ce-88c1-4055-8da7-1c1861f81a04 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:09.683271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:09.684537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:09.684537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:09.685510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:09.687183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:20:09.687602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:09.689380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:09.710283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0718760e-4a66-48a9-ae75-d2e74d9493cf None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:09.715949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 979/1952] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:20:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:20:09.720048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0718760e-4a66-48a9-ae75-d2e74d9493cf None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:09.725286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:09.725830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:09.726189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:09.726473 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:09.727228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:09.729358 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:09.858996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 974/1953] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:20:09 2022] POST /identity/v3/projects => generated 387 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:20:09.859155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:09.876729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e489f8f-cdb4-4401-8d35-1ed5df15a963 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:09.973798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:09.973798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:09.973798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:09.973798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:09.974442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:09.974736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:10.105615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 980/1954] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:20:09 2022] POST /identity/v3/users => generated 294 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:20:10.109067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:10.128511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633fa866-740e-4cb7-8ae8-707be84de094 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:10.140188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:10.141059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.141478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.142161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.142606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:10.146224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:10.157540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 975/1955] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:10 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:20:10.161241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:10.181286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b0004-32a3-4ef4-87a6-647c7f88703e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:10.181286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea20613-28c0-477b-b6c2-164ee7d84387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:10.182235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea20613-28c0-477b-b6c2-164ee7d84387 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.182409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea20613-28c0-477b-b6c2-164ee7d84387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.182966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea20613-28c0-477b-b6c2-164ee7d84387 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.183729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea20613-28c0-477b-b6c2-164ee7d84387 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:10.185798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea20613-28c0-477b-b6c2-164ee7d84387 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:10.205108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 981/1956] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:10 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:20:10.206391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:10.219654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2177ac34-6d5a-4963-9622-b74d9ca90f0a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:10.232885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:10.232885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.232885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.232885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] PATH_INFO: `/v3/projects/d4dd16a7faad40a6b12adb30f4535c64/users/708e54aad9574d28bb1ec8904bdf197f/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.267122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4dd16a7faad40a6b12adb30f4535c64, user_id=708e54aad9574d28bb1ec8904bdf197f, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:10.268917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:10.339594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:10.355220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 976/1957] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:20:10 2022] PUT /identity/v3/projects/d4dd16a7faad40a6b12adb30f4535c64/users/708e54aad9574d28bb1ec8904bdf197f/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:10.356492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043b51f-ddd1-495d-9801-34862ac941f8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.356849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043b51f-ddd1-495d-9801-34862ac941f8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.357134 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043b51f-ddd1-495d-9801-34862ac941f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.389616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5043b51f-ddd1-495d-9801-34862ac941f8 None None] MFA Rules not processed for user `708e54aad9574d28bb1ec8904bdf197f`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:10.391067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5043b51f-ddd1-495d-9801-34862ac941f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:10.492707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 982/1958] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:10 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:10.497191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:10.568983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115cafc9-2713-46b6-8237-80f490021506 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:10.573955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:10.574564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.574844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.575114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.576390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:10.578344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:10.676366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 977/1959] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:20:10 2022] POST /identity/v3/projects => generated 387 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:20:10.680310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043b51f-ddd1-495d-9801-34862ac941f8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:10.691997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043b51f-ddd1-495d-9801-34862ac941f8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:10.731432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:10.750441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:10.751059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.751341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.752027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.752705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:10.755020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:10.887320 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 983/1960] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:20:10 2022] POST /identity/v3/users => generated 300 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:20:10.891657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:10.902023 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83351740-129d-4c4a-ab7c-075353b87cb3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:10.906761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:10.907364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.907650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.907929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.908650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:10.914058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:10.924384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 978/1961] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:20:10.928640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:10.939295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d847b2-cd95-4114-b65e-8530718b8b6b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:10.943968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:10.944628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:10.944914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:10.945727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] PATH_INFO: `/v3/projects/f552c1d725f54ac7bd16d1f6d72d11bd/users/2bc55d1dc16e46a39488b5801eb94db6/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:10.960644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f552c1d725f54ac7bd16d1f6d72d11bd, user_id=2bc55d1dc16e46a39488b5801eb94db6, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:10.965487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:11.069042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 984/1962] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:20:10 2022] PUT /identity/v3/projects/f552c1d725f54ac7bd16d1f6d72d11bd/users/2bc55d1dc16e46a39488b5801eb94db6/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:11.069948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:11.081282 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c92f1f-4ab7-4001-b81b-c5bff039d297 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:11.144017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163daf56-b302-4610-b3d6-2eca42f0b9f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:11.144765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163daf56-b302-4610-b3d6-2eca42f0b9f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:11.145131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163daf56-b302-4610-b3d6-2eca42f0b9f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:11.145524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163daf56-b302-4610-b3d6-2eca42f0b9f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:11.146342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163daf56-b302-4610-b3d6-2eca42f0b9f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:11.148157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163daf56-b302-4610-b3d6-2eca42f0b9f1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:11.159569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 979/1963] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:11 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:20:11.164449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:11.176107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7855721-143f-4d86-988f-6a3d169377a9 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:11.180719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:11.181458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:11.181733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:11.182005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] PATH_INFO: `/v3/projects/f552c1d725f54ac7bd16d1f6d72d11bd/users/2bc55d1dc16e46a39488b5801eb94db6/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:11.183878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f552c1d725f54ac7bd16d1f6d72d11bd, user_id=2bc55d1dc16e46a39488b5801eb94db6, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:11.186797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:11.198366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 985/1964] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:20:11 2022] PUT /identity/v3/projects/f552c1d725f54ac7bd16d1f6d72d11bd/users/2bc55d1dc16e46a39488b5801eb94db6/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:11.274930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478444ba-b1a9-4fee-8c6b-9f483eb4d808 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:11.275228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478444ba-b1a9-4fee-8c6b-9f483eb4d808 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:11.275516 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478444ba-b1a9-4fee-8c6b-9f483eb4d808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:11.311521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-478444ba-b1a9-4fee-8c6b-9f483eb4d808 None None] MFA Rules not processed for user `2bc55d1dc16e46a39488b5801eb94db6`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:11.317069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-478444ba-b1a9-4fee-8c6b-9f483eb4d808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:11.495632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:11.519749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 980/1965] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:11 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:11.533436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:11.535290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:11.667459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ee0923-128f-4fc2-aeb6-bb89d1de8ba4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:11.671982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83537e2c-410f-4c09-9840-be8e891a9179 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:11.672618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83537e2c-410f-4c09-9840-be8e891a9179 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:11.672902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83537e2c-410f-4c09-9840-be8e891a9179 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:11.673233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83537e2c-410f-4c09-9840-be8e891a9179 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:11.684362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83537e2c-410f-4c09-9840-be8e891a9179 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:11.686142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83537e2c-410f-4c09-9840-be8e891a9179 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:11.697549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 986/1966] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:11 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:11.739910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478444ba-b1a9-4fee-8c6b-9f483eb4d808 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:11.809444 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478444ba-b1a9-4fee-8c6b-9f483eb4d808 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:11.814828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:11.815477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:11.815763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:11.816044 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] PATH_INFO: `/v3/users/708e54aad9574d28bb1ec8904bdf197f` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:11.817167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=708e54aad9574d28bb1ec8904bdf197f)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:11.819158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:12.031420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '708e54aad9574d28bb1ec8904bdf197f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:12.064353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '708e54aad9574d28bb1ec8904bdf197f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:12.149658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '708e54aad9574d28bb1ec8904bdf197f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:12.150294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '708e54aad9574d28bb1ec8904bdf197f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:12.150910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '708e54aad9574d28bb1ec8904bdf197f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:12.168087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 981/1967] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:20:11 2022] DELETE /identity/v3/users/708e54aad9574d28bb1ec8904bdf197f => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:12.172344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:12.172634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:12.172907 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:12.193935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:12.195248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:12.259806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 987/1968] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:12.270532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:12.272018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:12.296928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:12.298619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:12.336494 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5328f2de-217a-418c-9ce2-8285776710bf admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:12.340652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:12.342299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:12.342658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:12.343027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:12.348057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:12.352707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:12.354473 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:12.364737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 982/1969] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:12.378034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bae4976-f0a2-45b2-afbc-87a9644c5b36 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:12.383178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:12.383806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:12.384072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:12.384344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:12.386050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:12.400105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:12.401822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:12.417517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 988/1970] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:20:12.932868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:12.934387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:12.960252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e4896f-6b53-4392-bf64-d55b5c85c071 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:12.969623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:12.970300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:12.977685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:12.977685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] PATH_INFO: `/v3/projects/d4dd16a7faad40a6b12adb30f4535c64` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:12.977685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4dd16a7faad40a6b12adb30f4535c64)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:12.977685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:12.986702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:13.094952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4dd16a7faad40a6b12adb30f4535c64', 'request_id': 'req-76545871-012d-4e54-b6fd-598459eb60c3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.150025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4dd16a7faad40a6b12adb30f4535c64', 'request_id': 'req-76545871-012d-4e54-b6fd-598459eb60c3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.161584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4dd16a7faad40a6b12adb30f4535c64', 'request_id': 'req-76545871-012d-4e54-b6fd-598459eb60c3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.169920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] The token cache is being invalidate because project d4dd16a7faad40a6b12adb30f4535c64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:20:13.170703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.171083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.172659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 983/1971] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:20:12 2022] DELETE /identity/v3/projects/d4dd16a7faad40a6b12adb30f4535c64 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:13.176861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:13.190018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1654ee17-3702-4fa7-9d6b-fe23228d7644 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:13.255124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:13.255948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:13.256292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:13.256943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] PATH_INFO: `/v3/users/2bc55d1dc16e46a39488b5801eb94db6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:13.258291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bc55d1dc16e46a39488b5801eb94db6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:13.260270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:13.471069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bc55d1dc16e46a39488b5801eb94db6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.471777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bc55d1dc16e46a39488b5801eb94db6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.484418 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bc55d1dc16e46a39488b5801eb94db6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.491379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bc55d1dc16e46a39488b5801eb94db6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.492088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a235ca1c-6b84-494b-a290-4df6c7ea7602 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bc55d1dc16e46a39488b5801eb94db6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:13.578640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 989/1972] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:20:13 2022] DELETE /identity/v3/users/2bc55d1dc16e46a39488b5801eb94db6 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:14.116459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:14.117985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:14.133188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76545871-012d-4e54-b6fd-598459eb60c3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:14.238359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:14.239028 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:14.239343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:14.239576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] PATH_INFO: `/v3/projects/f552c1d725f54ac7bd16d1f6d72d11bd` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:14.241359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f552c1d725f54ac7bd16d1f6d72d11bd)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:14.243525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:14.375668 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f552c1d725f54ac7bd16d1f6d72d11bd', 'request_id': 'req-5520818c-d7aa-45de-9d15-66ebf64ab3a7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:14.423979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f552c1d725f54ac7bd16d1f6d72d11bd', 'request_id': 'req-5520818c-d7aa-45de-9d15-66ebf64ab3a7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:14.432194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f552c1d725f54ac7bd16d1f6d72d11bd', 'request_id': 'req-5520818c-d7aa-45de-9d15-66ebf64ab3a7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:14.445959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] The token cache is being invalidate because project f552c1d725f54ac7bd16d1f6d72d11bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:20:14.446243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:14.447442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5520818c-d7aa-45de-9d15-66ebf64ab3a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:14.449193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 984/1973] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:20:14 2022] DELETE /identity/v3/projects/f552c1d725f54ac7bd16d1f6d72d11bd => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:14.454947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c35fa5-7cd5-47c9-b51a-e53f28f5d795 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:14.455836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c35fa5-7cd5-47c9-b51a-e53f28f5d795 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:14.456298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c35fa5-7cd5-47c9-b51a-e53f28f5d795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:14.482428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-56c35fa5-7cd5-47c9-b51a-e53f28f5d795 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:14.484287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-56c35fa5-7cd5-47c9-b51a-e53f28f5d795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:14.533397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:14.553430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 990/1974] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:14.561798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432936c-4722-4d61-a742-8e9ded644088 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:14.562281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432936c-4722-4d61-a742-8e9ded644088 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:14.562779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432936c-4722-4d61-a742-8e9ded644088 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:14.600450 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c432936c-4722-4d61-a742-8e9ded644088 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:14.601900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c432936c-4722-4d61-a742-8e9ded644088 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:14.616211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 985/1975] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:14.616426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c35fa5-7cd5-47c9-b51a-e53f28f5d795 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:14.626603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c35fa5-7cd5-47c9-b51a-e53f28f5d795 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:14.632283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:14.632903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:14.633186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:14.633528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:14.634349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:20:14.634663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:14.636287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:14.652200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 991/1976] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:20:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:20:14.660972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c432936c-4722-4d61-a742-8e9ded644088 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:14.671346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c432936c-4722-4d61-a742-8e9ded644088 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:14.676400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:14.676954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:14.677230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:14.677556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:14.678185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:14.680137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:14.828141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 986/1977] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:20:14 2022] POST /identity/v3/projects => generated 401 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:20:14.832600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:14.844601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f591d1-d581-4103-9f70-f5d667ef6ad6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:14.921214 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:14.922179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:14.922621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:14.923082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:14.923942 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:14.927584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.031177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 992/1978] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:20:14 2022] POST /identity/v3/users => generated 301 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:20:15.035051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.040754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:15.046695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3861e4-c4e1-41fc-8bf6-a267efcebba4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:15.052297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:15.052998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.053295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.055966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.055966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:15.055966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.067263 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 987/1979] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:20:15.071641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.099889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6881d4-8800-4694-80fb-0e0c0fa46d46 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:15.108562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bb92ba-e3d3-4794-a65d-a88980738409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:15.109197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bb92ba-e3d3-4794-a65d-a88980738409 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.109508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bb92ba-e3d3-4794-a65d-a88980738409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.109781 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bb92ba-e3d3-4794-a65d-a88980738409 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.110420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bb92ba-e3d3-4794-a65d-a88980738409 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:15.112031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bb92ba-e3d3-4794-a65d-a88980738409 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.127056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 993/1980] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:20:15.131035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.148170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec3ed44-1872-4920-97b9-ba704d83004b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:15.154023 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:15.155039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.155547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.156087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] PATH_INFO: `/v3/projects/1d925a3ac6df4c4680e3fa6f3fa25a71/users/30de2f55c8c3476ea184232b6de87850/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.178926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d925a3ac6df4c4680e3fa6f3fa25a71, user_id=30de2f55c8c3476ea184232b6de87850, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:15.182452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.247712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 988/1981] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:20:15 2022] PUT /identity/v3/projects/1d925a3ac6df4c4680e3fa6f3fa25a71/users/30de2f55c8c3476ea184232b6de87850/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:15.258319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc71d90-fdfa-4c66-b709-b7f3db81aa97 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.258610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc71d90-fdfa-4c66-b709-b7f3db81aa97 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.258886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc71d90-fdfa-4c66-b709-b7f3db81aa97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.303166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5fc71d90-fdfa-4c66-b709-b7f3db81aa97 None None] MFA Rules not processed for user `30de2f55c8c3476ea184232b6de87850`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:15.305228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5fc71d90-fdfa-4c66-b709-b7f3db81aa97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:15.410390 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 994/1982] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:15 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:20:15.414832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.486664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a50abf-e7a1-4937-847c-8b695cc3d484 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:15.493165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:15.493917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.494298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.494886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.495467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:15.497830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.582879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 989/1983] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:20:15 2022] POST /identity/v3/projects => generated 399 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:20:15.584189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc71d90-fdfa-4c66-b709-b7f3db81aa97 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.597717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc71d90-fdfa-4c66-b709-b7f3db81aa97 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:15.640732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:15.662263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:15.662942 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.663279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.663619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.666685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:15.668789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.780920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 995/1984] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:20:15 2022] POST /identity/v3/users => generated 306 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:20:15.785199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.800687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d120b5-bf90-4d2d-af3b-1c48e4f2fb89 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:15.809210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:15.810537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.810816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.811117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.811742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:15.813376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.829595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 990/1985] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:20:15.835825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.850892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f894ccd-388a-4b11-9061-09d1f6a5fd31 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:15.856625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:15.858866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:15.859153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:15.859440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] PATH_INFO: `/v3/projects/62cf15b7a3b648ef941c132f0b0977db/users/8518175c00f74cdd9d8493ddca321f2b/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:15.885616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62cf15b7a3b648ef941c132f0b0977db, user_id=8518175c00f74cdd9d8493ddca321f2b, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:15.888903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:15.961435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 996/1986] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:20:15 2022] PUT /identity/v3/projects/62cf15b7a3b648ef941c132f0b0977db/users/8518175c00f74cdd9d8493ddca321f2b/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:15.965656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:15.976369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a991f-be3a-4b8e-a3f7-c8931a5aedfd admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:16.046537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3161b47-9576-4568-8455-02e92f6db4f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:16.047159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3161b47-9576-4568-8455-02e92f6db4f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:16.050375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3161b47-9576-4568-8455-02e92f6db4f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:16.050375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3161b47-9576-4568-8455-02e92f6db4f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:16.050375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3161b47-9576-4568-8455-02e92f6db4f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:16.050375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3161b47-9576-4568-8455-02e92f6db4f6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:16.060791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 991/1987] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:20:16.065476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:16.076608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9dce8-7801-4ba7-a670-582b9e646de6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:16.081276 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:16.081974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:16.082258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:16.082688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] PATH_INFO: `/v3/projects/62cf15b7a3b648ef941c132f0b0977db/users/8518175c00f74cdd9d8493ddca321f2b/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:16.084606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62cf15b7a3b648ef941c132f0b0977db, user_id=8518175c00f74cdd9d8493ddca321f2b, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:16.087646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:16.098890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 997/1988] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:20:16 2022] PUT /identity/v3/projects/62cf15b7a3b648ef941c132f0b0977db/users/8518175c00f74cdd9d8493ddca321f2b/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:16.110959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17191354-f528-45b4-a088-ef6b3a1f1839 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:16.111360 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17191354-f528-45b4-a088-ef6b3a1f1839 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:16.111571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17191354-f528-45b4-a088-ef6b3a1f1839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:16.141918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-17191354-f528-45b4-a088-ef6b3a1f1839 None None] MFA Rules not processed for user `8518175c00f74cdd9d8493ddca321f2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:20:16.143147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-17191354-f528-45b4-a088-ef6b3a1f1839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:16.173557 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:16.220177 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 992/1989] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:20:16 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:20:16.230783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:16.232515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:16.299654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ec3f1c-f5ef-491b-b58d-a72542f87f11 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:16.303958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:16.304270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:16.304270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:16.304610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:16.317937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:16.319415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:16.336232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 998/1990] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:16 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:16.673865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17191354-f528-45b4-a088-ef6b3a1f1839 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:16.684648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17191354-f528-45b4-a088-ef6b3a1f1839 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:16.689098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:16.689895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:16.690314 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:16.690600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:16.780337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:16.781962 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:16.791804 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 993/1991] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:16 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:17.075175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:17.082031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:17.154039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9883778-966a-4cbb-815b-26ef9687bbdf service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:17.159337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:17.159913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:17.160190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:17.160461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:17.162228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:17.173749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:17.175315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:17.185687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 999/1992] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:17.203453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:17.203453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:17.283289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb2be9-12b8-4d32-a237-78baf3310fde service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:17.287780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:17.288446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:17.293414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:17.293414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] PATH_INFO: `/v3/users/a5c784f29d2e4cb4bb8bd6d62fbc2216` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:17.303601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5c784f29d2e4cb4bb8bd6d62fbc2216)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:17.305560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:17.465592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5c784f29d2e4cb4bb8bd6d62fbc2216'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:17.471085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:17.479564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5c784f29d2e4cb4bb8bd6d62fbc2216'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:17.573283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5c784f29d2e4cb4bb8bd6d62fbc2216'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:17.573940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5c784f29d2e4cb4bb8bd6d62fbc2216'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:17.574542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5c784f29d2e4cb4bb8bd6d62fbc2216'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:17.588277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 994/1993] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:20:17 2022] DELETE /identity/v3/users/a5c784f29d2e4cb4bb8bd6d62fbc2216 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:17.819277 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:17.833841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ca7f58-db09-44e9-8d31-f6f9e6ce8e1a service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:17.857565 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:17.896848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:17.897526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:17.897807 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:17.898084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] PATH_INFO: `/v3/projects/fac1485ccdeb434ca49caa52ebfdc24b` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:17.899242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fac1485ccdeb434ca49caa52ebfdc24b)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:17.901195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:18.003278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fac1485ccdeb434ca49caa52ebfdc24b', 'request_id': 'req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.009181 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fac1485ccdeb434ca49caa52ebfdc24b', 'request_id': 'req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.022823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fac1485ccdeb434ca49caa52ebfdc24b', 'request_id': 'req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.093604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] The token cache is being invalidate because project fac1485ccdeb434ca49caa52ebfdc24b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:20:18.094174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.094561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.097714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1000/1994] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:20:17 2022] DELETE /identity/v3/projects/fac1485ccdeb434ca49caa52ebfdc24b => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:18.100416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:18.101849 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:18.115784 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6627f9eb-ca8e-4566-addb-ad98c4cc58b1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:18.176082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:18.176729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:18.177009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:18.177283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] PATH_INFO: `/v3/users/a113313fcf584b409de2da73424c1386` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:18.191199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] RBAC: Authorizing `identity:delete_user(user_id=a113313fcf584b409de2da73424c1386)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:18.193164 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:18.402938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a113313fcf584b409de2da73424c1386'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.415017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a113313fcf584b409de2da73424c1386'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.480707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a113313fcf584b409de2da73424c1386'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.481361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a113313fcf584b409de2da73424c1386'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.482506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a113313fcf584b409de2da73424c1386'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:18.496984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 995/1995] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:20:18 2022] DELETE /identity/v3/users/a113313fcf584b409de2da73424c1386 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:20:18.850584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:18.851404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:18.865669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3b6c3b-0e4f-49a0-a0fe-81fdb3233d82 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:18.926148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:18.926711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:18.926994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:18.927272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] PATH_INFO: `/v3/projects/9ab6cbbf0c2c43388ed1a33f737a2951` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:18.938038 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ab6cbbf0c2c43388ed1a33f737a2951)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:18.942399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:19.035625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:19.036979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ab6cbbf0c2c43388ed1a33f737a2951', 'request_id': 'req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:19.042641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ab6cbbf0c2c43388ed1a33f737a2951', 'request_id': 'req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:19.049446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ab6cbbf0c2c43388ed1a33f737a2951', 'request_id': 'req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:19.126476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] The token cache is being invalidate because project 9ab6cbbf0c2c43388ed1a33f737a2951 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:20:19.127128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:19.127732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:20:19.129508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1001/1996] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:20:18 2022] DELETE /identity/v3/projects/9ab6cbbf0c2c43388ed1a33f737a2951 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:20:25.188856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:25.190372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:25.232998 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:25.248300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a3d36-4079-4d59-9214-609620c046da admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:25.296658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:25.297272 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:25.297588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:25.297866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:25.299609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:25.369243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:25.370712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:25.382074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 996/1997] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:28.795232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:28.796772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:28.808878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4b88e-d8d1-4c8a-a774-0823f8c3e17c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:28.813429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:28.814027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:28.814323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:28.817441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:28.817441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:28.830618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:28.832096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:28.845465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1002/1998] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:34.537855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:34.539239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:34.595228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cd7562-2c2c-40fd-ab17-a5da6950b598 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:34.598695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:34.599337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:34.599617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:34.599893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:34.601597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:34.654697 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:34.678832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:34.680330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:34.690873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 997/1999] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:34 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:38.838601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=13,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:38.913290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:38.914686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:38.928892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284f4114-936a-497e-8b97-a007d87c3981 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:38.935839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:38.936452 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:38.938788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:38.938788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:38.939174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:39.014978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:39.016617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:39.025898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1003/2000] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:38 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:41.928776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:20:41.942125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6380d0e-2ba4-4204-8c72-204cf23c9a07 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:20:41.947586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:20:41.948211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:20:41.948493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:20:41.948772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:20:41.951377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:20:42.030158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:20:42.031778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:20:42.041674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 998/2001] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:20:41 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:20:49.021604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=33 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:20:52.036856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=43 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:21:13.394514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:21:13.397831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:21:13.408200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7750c9-c08c-4b1b-a1ba-321a5e76f614 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:21:13.411816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:21:13.412419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:21:13.412705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:21:13.412983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:21:13.414684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:21:13.473745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:21:13.475933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:21:13.486229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1004/2002] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:21:13 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:21:23.480119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=28 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:21:41.917482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:21:41.918999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:21:41.931244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae277eed-9f44-4c28-83e3-e1121c501268 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:21:41.936543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:21:41.937805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:21:41.938086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:21:41.938406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:21:41.940107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:21:41.951857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:21:41.953606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:21:41.961681 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 999/2003] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:21:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:21:51.178041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:21:51.179470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:21:51.195631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baf00a9-b22e-4143-a26f-9315d0b2e674 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:21:51.201378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:21:51.201983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:21:51.202261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:21:51.202531 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:21:51.204894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:21:51.337889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:21:51.340112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:21:51.351959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1005/2004] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:21:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:21:51.810856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:21:51.828218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4061ba55-c3a2-4adf-be4d-869a421a0e44 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:21:51.835755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:21:51.836906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:21:51.837134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:21:51.837609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:21:51.839977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:21:51.860606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:21:51.862379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:21:51.881032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1000/2005] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:21:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:22:01.346462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=33 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:22:01.870134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=12 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:24:44.172864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:24:44.174332 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:24:44.311795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b69bd0-ca45-42f4-b012-05616fae80b1 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:24:44.317599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:24:44.318209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:24:44.318489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:24:44.318762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:24:44.320484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:24:44.331688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:24:44.333422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:24:44.343356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1006/2006] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:24:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:24:54.339051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=63 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:25:24.486448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:25:24.523747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7836b2ac-602f-4efb-b548-873990c43032 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:25:24.527427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:25:24.528032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:25:24.528318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:25:24.528599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:25:24.545622 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:25:24.547042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:25:24.567458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1001/2007] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:25:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:25:27.990788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:25:27.999999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7abde30-4d24-40a8-8a8f-743242c56b16 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:25:28.003586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:25:28.004178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:25:28.004461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:25:28.004735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:25:28.016109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:25:28.017570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:25:28.029929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1007/2008] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:25:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:25:33.518273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:25:33.530322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65627ab-e4cb-40d9-9319-9abffcb3da5a service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:25:33.536037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:25:33.538181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:25:33.538181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:25:33.538181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:25:33.546038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:25:33.547620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:25:33.556948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1002/2009] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:25:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:25:38.024258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:25:43.552489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=31 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:26:48.715450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:26:48.724285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e182e55-aa38-4c34-8707-afd9a0040fa7 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:26:48.728347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:26:48.728935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:26:48.729212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:26:48.729511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:26:48.736562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:26:48.737998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:26:48.745785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1008/2010] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:26:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 31 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:26:58.742290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:27:00.025889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:00.045972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fd8d6-8454-466f-8140-288a1cfe6e21 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:00.049780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:00.050390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:00.050669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:00.050949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:00.059832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:00.064935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:00.090967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1003/2011] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:00 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:27:00.135502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:00.144511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2010534-fa3a-45fa-8903-33232c99142e service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:00.148053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:00.148628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:00.148906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:00.149188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:00.158046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:00.159453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:00.186042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1009/2012] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:00 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:27:00.228959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:00.237266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d1014-31f5-473e-a4ea-0ae3f1a4578a service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:00.240922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:00.241559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:00.241839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:00.242110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:00.252137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:00.253600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:00.281114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1004/2013] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:00 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:27:00.330013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:00.338437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88a9cd-9d34-4ead-8795-51dd490013ac service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:00.342044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852dbcba-fffb-415e-be33-4cb9965f2ea7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:00.342637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852dbcba-fffb-415e-be33-4cb9965f2ea7 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:00.342920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852dbcba-fffb-415e-be33-4cb9965f2ea7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:00.343190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852dbcba-fffb-415e-be33-4cb9965f2ea7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:00.345159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-852dbcba-fffb-415e-be33-4cb9965f2ea7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:27:00.404112 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852dbcba-fffb-415e-be33-4cb9965f2ea7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:00.405825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852dbcba-fffb-415e-be33-4cb9965f2ea7 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:00.414617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1010/2014] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:00 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:27:00.603646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:00.612031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2813c802-41e0-48df-a13e-3459d6ee5d95 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:00.616017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:00.616713 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:00.617008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:00.617288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:00.626723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:00.628150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:00.656132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1005/2015] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:27:10.410632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=41 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:27:10.653118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=43 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:27:51.988514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:51.992645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:51.992645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:52.027526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] MFA Rules not processed for user `724baaa4aa834e609fe4f513f9fc0b4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:27:52.028728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:27:52.045048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1011/2016] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:27:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:27:52.344964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:52.355079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987de337-5c2f-4269-93b4-4571305acfde service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:52.358717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:52.359316 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:52.359595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:52.359868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:52.361579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:27:52.383580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:52.384979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:52.393474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1006/2017] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:27:52.799991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:52.801394 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:27:52.817021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bc15d5-1d82-443f-a6cb-8d3ab00e0b69 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:52.822004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:52.826739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:52.826739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:52.826739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:52.826739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:27:52.837628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:52.839278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:52.850323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1012/2018] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:27:56.989482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:27:57.003089 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded93ff0-47f5-4783-aba3-4faf56e95254 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:27:57.008657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:27:57.009452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:27:57.011946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:27:57.011946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:27:57.020664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:27:57.022299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:27:57.035488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1007/2019] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:27:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:28:02.144510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:28:02.157719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c28ea1-c7d3-438a-b97b-2c0f8094f080 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:28:02.163117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:28:02.163715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:28:02.163994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:28:02.164270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:28:02.165998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:28:02.190921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:28:02.192435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:28:02.202941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1013/2020] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:28:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:28:05.790193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:28:05.802010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b8dcdc-e574-45ee-ba27-3a7b0529f503 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:28:05.807630 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:28:05.808234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:28:05.808515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:28:05.808789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:28:05.810555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:28:05.823279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:28:05.824943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:28:05.836355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1008/2021] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:28:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:28:12.198516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=31 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:28:15.831434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=23 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:29:01.069629 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:29:01.079022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3bc603-4406-4789-8388-bb319767b3d5 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:29:01.082614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:29:01.083211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:29:01.083492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:29:01.083763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:29:01.085700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:29:01.147544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:29:01.148966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:29:01.158322 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1014/2022] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:29:01 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:29:11.154131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=30 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:29:47.198991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:29:47.210170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafaf7da-07ea-4b5b-84fb-9c3deeec1e81 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:29:47.215156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:29:47.215897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:29:47.216189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:29:47.216466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:29:47.225064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:29:47.226663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:29:47.237263 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1009/2023] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:29:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:29:57.232984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:30:24.360991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:30:24.370159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0354cf7-46cb-4f57-9e9c-2fc2a41866d0 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:30:24.373698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:30:24.374369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:30:24.374648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:30:24.374920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:30:24.384729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:30:24.386204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:30:24.397637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1015/2024] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:30:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:30:30.275525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:30:30.277223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:30:30.336499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee0b8be-655c-4197-9e41-715399edad8c service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:30:30.340132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:30:30.340738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:30:30.341021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:30:30.341295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:30:30.343744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:30:30.414366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:30:30.415783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:30:30.426676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1010/2025] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:30:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:30:34.392917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:30:40.421923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=54 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:01.818784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:01.875778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0898567-3f00-42fd-9d81-2a5a5e672612 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:01.879478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fa9942-59e3-4172-b957-dfc556f8bded service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:01.880082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fa9942-59e3-4172-b957-dfc556f8bded service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:01.880362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fa9942-59e3-4172-b957-dfc556f8bded service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:01.880634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fa9942-59e3-4172-b957-dfc556f8bded service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:01.882622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-81fa9942-59e3-4172-b957-dfc556f8bded service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:01.954868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fa9942-59e3-4172-b957-dfc556f8bded service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:01.959019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fa9942-59e3-4172-b957-dfc556f8bded service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:01.970192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1016/2026] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:31:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:31:02.179290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:02.180782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:02.332237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68051246-cf9c-49e7-b6ee-5ca7d68e9bde service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:02.340737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:02.341439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:02.341729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:02.342012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] PATH_INFO: `/v3/users/30de2f55c8c3476ea184232b6de87850` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:02.365177 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=30de2f55c8c3476ea184232b6de87850)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:02.367853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:02.550919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30de2f55c8c3476ea184232b6de87850'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:02.567591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30de2f55c8c3476ea184232b6de87850'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:02.621623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30de2f55c8c3476ea184232b6de87850'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:02.622120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30de2f55c8c3476ea184232b6de87850'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:02.622742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30de2f55c8c3476ea184232b6de87850'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:02.642249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1011/2027] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:31:02 2022] DELETE /identity/v3/users/30de2f55c8c3476ea184232b6de87850 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:31:02.646722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:02.647120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:02.647614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:02.680555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:31:02.682514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:02.746534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1017/2028] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:31:02.758046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:02.759458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:02.808055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:02.824460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81638296-61d3-4b20-8ef8-1d9ef7e438dc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:02.830374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:02.831098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:02.831378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:02.831654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:02.843686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:02.848045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:02.861334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1012/2029] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:31:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:31:03.280211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:03.282753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:03.294434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e6dddd-d224-4349-b6a7-1588d9e42ae6 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:03.299923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:03.300775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:03.301065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:03.301388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] PATH_INFO: `/v3/projects/1d925a3ac6df4c4680e3fa6f3fa25a71` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:03.312954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d925a3ac6df4c4680e3fa6f3fa25a71)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:03.314939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:03.341489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:03.467999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d925a3ac6df4c4680e3fa6f3fa25a71', 'request_id': 'req-06114f84-4950-48da-a01f-39dc329a7ae2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.474318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d925a3ac6df4c4680e3fa6f3fa25a71', 'request_id': 'req-06114f84-4950-48da-a01f-39dc329a7ae2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.482064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d925a3ac6df4c4680e3fa6f3fa25a71', 'request_id': 'req-06114f84-4950-48da-a01f-39dc329a7ae2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.560564 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] The token cache is being invalidate because project 1d925a3ac6df4c4680e3fa6f3fa25a71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:31:03.560875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.564730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.569971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1018/2030] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:31:03 2022] DELETE /identity/v3/projects/1d925a3ac6df4c4680e3fa6f3fa25a71 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:31:03.574181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:03.585175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370fc48-b6e5-48cb-a5da-55308f93f355 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:03.642287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:03.642934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:03.643219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:03.643498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] PATH_INFO: `/v3/users/8518175c00f74cdd9d8493ddca321f2b` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:03.656699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=8518175c00f74cdd9d8493ddca321f2b)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:03.658706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:03.818334 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8518175c00f74cdd9d8493ddca321f2b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.830637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8518175c00f74cdd9d8493ddca321f2b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.887768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8518175c00f74cdd9d8493ddca321f2b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.888413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8518175c00f74cdd9d8493ddca321f2b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.889022 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6ef5528e-d73c-485c-9f10-12aa2ca2e7eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8518175c00f74cdd9d8493ddca321f2b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:03.901621 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1013/2031] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:31:03 2022] DELETE /identity/v3/users/8518175c00f74cdd9d8493ddca321f2b => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:31:04.320634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:04.322555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:04.335821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06114f84-4950-48da-a01f-39dc329a7ae2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:04.392982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:04.393668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:04.399340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:04.399340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] PATH_INFO: `/v3/projects/62cf15b7a3b648ef941c132f0b0977db` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:04.404444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62cf15b7a3b648ef941c132f0b0977db)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:04.406411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:04.512782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62cf15b7a3b648ef941c132f0b0977db', 'request_id': 'req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:04.518916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62cf15b7a3b648ef941c132f0b0977db', 'request_id': 'req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:04.525664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62cf15b7a3b648ef941c132f0b0977db', 'request_id': 'req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:04.609191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] The token cache is being invalidate because project 62cf15b7a3b648ef941c132f0b0977db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:31:04.609524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:04.610143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3bfa259b-8e6b-4f1d-a295-7a9d7cc7e532 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:31:04.611688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1019/2032] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:31:04 2022] DELETE /identity/v3/projects/62cf15b7a3b648ef941c132f0b0977db => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:31:04.618382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843ee4d-a23e-4dec-b801-70b1f44e3f8a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:04.618884 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843ee4d-a23e-4dec-b801-70b1f44e3f8a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:04.619098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843ee4d-a23e-4dec-b801-70b1f44e3f8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:04.672214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9843ee4d-a23e-4dec-b801-70b1f44e3f8a None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:31:04.673631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9843ee4d-a23e-4dec-b801-70b1f44e3f8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:04.695680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:04.737235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1014/2033] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:31:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:31:04.745279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60572e-3427-4647-859e-09259d72d85b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:04.745598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60572e-3427-4647-859e-09259d72d85b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:04.745876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60572e-3427-4647-859e-09259d72d85b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:04.766387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-9b60572e-3427-4647-859e-09259d72d85b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:31:04.768727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9b60572e-3427-4647-859e-09259d72d85b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:04.776858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1020/2034] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:31:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:31:04.781090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843ee4d-a23e-4dec-b801-70b1f44e3f8a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:04.791834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843ee4d-a23e-4dec-b801-70b1f44e3f8a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:04.798646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:04.798646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:04.798646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:04.798646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:04.799132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:31:04.799132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:04.800574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:04.811970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1015/2035] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:31:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:31:04.816696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b60572e-3427-4647-859e-09259d72d85b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:04.825970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b60572e-3427-4647-859e-09259d72d85b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:04.831370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:04.831969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:04.832257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:04.832532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:04.833162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:04.835100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:04.861749 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:04.960334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1021/2036] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:31:04 2022] POST /identity/v3/projects => generated 371 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:31:04.964476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:04.975004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e59fac-0dfa-42b8-8de6-2b025a38c930 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.033744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.034368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.034657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.034939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.035588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.037658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.159147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1016/2037] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:31:04 2022] POST /identity/v3/users => generated 286 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:31:05.163283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.174068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0124e8ef-9285-4896-92c7-90374d3427ad admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.178589 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.179182 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.179465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.179737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.180440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.182058 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.192413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1022/2038] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:31:05.196216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.206843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400a1221-28e3-491b-8915-206db9ed1fc7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.211489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aba9b88-f02b-4be7-9c49-5615d292dcb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.212090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aba9b88-f02b-4be7-9c49-5615d292dcb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.212376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aba9b88-f02b-4be7-9c49-5615d292dcb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.212656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aba9b88-f02b-4be7-9c49-5615d292dcb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.213294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aba9b88-f02b-4be7-9c49-5615d292dcb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.214920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aba9b88-f02b-4be7-9c49-5615d292dcb6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.225911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1017/2039] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:31:05.230146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.241441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481f8077-5d68-48d6-971d-8c30495e310d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.246097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.246730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.247018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.247300 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] PATH_INFO: `/v3/projects/b912c2edc2394dafbb0a9855416a6b19/users/a6c2a143a5d24b83998acbbd1df2993a/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.262894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b912c2edc2394dafbb0a9855416a6b19, user_id=a6c2a143a5d24b83998acbbd1df2993a, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.265634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.358726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1023/2040] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:31:05 2022] PUT /identity/v3/projects/b912c2edc2394dafbb0a9855416a6b19/users/a6c2a143a5d24b83998acbbd1df2993a/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:31:05.370003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7b977f-3144-46b3-912a-79c93eb01aaa None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.370297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7b977f-3144-46b3-912a-79c93eb01aaa None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.370577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7b977f-3144-46b3-912a-79c93eb01aaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.399709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7c7b977f-3144-46b3-912a-79c93eb01aaa None None] MFA Rules not processed for user `a6c2a143a5d24b83998acbbd1df2993a`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:31:05.401303 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c7b977f-3144-46b3-912a-79c93eb01aaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:05.468329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1018/2041] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:31:05 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:31:05.473082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.535962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c1954-85a2-4c88-ae54-e9885c008707 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.540650 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.541265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.541582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.541863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.542496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.544405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.662121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1024/2042] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:31:05 2022] POST /identity/v3/projects => generated 373 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:31:05.666318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7b977f-3144-46b3-912a-79c93eb01aaa None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.677793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7b977f-3144-46b3-912a-79c93eb01aaa None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.696778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:05.734344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.735192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.735653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.735936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.737258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.739275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.842008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1019/2043] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:31:05 2022] POST /identity/v3/users => generated 293 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:31:05.843163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.854320 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41f445-632c-4164-9ae6-bb8e647742dd admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.859012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.859608 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.859890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.860168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.862613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.864466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.877450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1025/2044] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:31:05.881256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.893394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d06498-857f-4b89-ac26-45c30edcb621 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:05.898382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:05.899139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:05.899433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:05.899721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] PATH_INFO: `/v3/projects/f9b02b8bcf9e478bb4c8bc71bbb9b8ad/users/7914e923c7f7439ab5c2c07369a4a4b8/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:05.916014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9b02b8bcf9e478bb4c8bc71bbb9b8ad, user_id=7914e923c7f7439ab5c2c07369a4a4b8, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:05.918745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:05.981948 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1020/2045] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:31:05 2022] PUT /identity/v3/projects/f9b02b8bcf9e478bb4c8bc71bbb9b8ad/users/7914e923c7f7439ab5c2c07369a4a4b8/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:31:05.986231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:05.996401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e3d6a-ef17-4afd-b089-cd6760d5ab8d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:06.055474 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9045b0-4f6e-4514-83ba-b53576352875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:06.056092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9045b0-4f6e-4514-83ba-b53576352875 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:06.056378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9045b0-4f6e-4514-83ba-b53576352875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:06.056654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9045b0-4f6e-4514-83ba-b53576352875 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:06.057291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9045b0-4f6e-4514-83ba-b53576352875 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:06.059026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9045b0-4f6e-4514-83ba-b53576352875 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:06.069042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1026/2046] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:31:06.073227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:06.084264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7351bc-910c-465b-af9d-66bf79e8a272 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:06.088984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:06.089679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:06.089967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:06.090248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] PATH_INFO: `/v3/projects/f9b02b8bcf9e478bb4c8bc71bbb9b8ad/users/7914e923c7f7439ab5c2c07369a4a4b8/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:06.092227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9b02b8bcf9e478bb4c8bc71bbb9b8ad, user_id=7914e923c7f7439ab5c2c07369a4a4b8, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:06.094903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:06.105455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1021/2047] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:31:06 2022] PUT /identity/v3/projects/f9b02b8bcf9e478bb4c8bc71bbb9b8ad/users/7914e923c7f7439ab5c2c07369a4a4b8/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:31:06.117569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:06.117861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:06.118142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:06.148848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] MFA Rules not processed for user `7914e923c7f7439ab5c2c07369a4a4b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:31:06.150089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:06.169198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:06.218859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1027/2048] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:31:06 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:31:16.101642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=34,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:16.266526 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=20 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:31:54.671419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:54.673129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:54.731414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e535b3-cebf-404a-a9df-5a77399ec1b7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:54.735023 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:54.735637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:54.735923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:54.736199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:54.738749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:54.748182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:54.749627 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:54.758184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1022/2049] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:31:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:31:59.716662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:31:59.718111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:59.790838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f9c6b3-6ab8-428b-826a-e480e1b95ea9 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:31:59.796465 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e509cf-f038-4dca-bc81-60efb96f55a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:31:59.797073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e509cf-f038-4dca-bc81-60efb96f55a1 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:31:59.797427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e509cf-f038-4dca-bc81-60efb96f55a1 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:31:59.797709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e509cf-f038-4dca-bc81-60efb96f55a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:31:59.799434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-62e509cf-f038-4dca-bc81-60efb96f55a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:31:59.870151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e509cf-f038-4dca-bc81-60efb96f55a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:31:59.871771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e509cf-f038-4dca-bc81-60efb96f55a1 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:31:59.881294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1028/2050] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:31:59 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:32:04.754747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=26 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:32:06.285376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:32:06.347850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d57629-cab7-43f3-ba3f-6ac30e8eae96 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:32:06.352772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:32:06.353449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:32:06.353742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:32:06.354026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] PATH_INFO: `/v3/users/a6c2a143a5d24b83998acbbd1df2993a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:32:06.355193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6c2a143a5d24b83998acbbd1df2993a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:32:06.357121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:32:06.544065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6c2a143a5d24b83998acbbd1df2993a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:06.556538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6c2a143a5d24b83998acbbd1df2993a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:06.626838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6c2a143a5d24b83998acbbd1df2993a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:06.627481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6c2a143a5d24b83998acbbd1df2993a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:06.628082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6c2a143a5d24b83998acbbd1df2993a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:06.641071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1023/2051] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:32:06 2022] DELETE /identity/v3/users/a6c2a143a5d24b83998acbbd1df2993a => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:32:06.646313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:32:06.649095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:32:06.649095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:32:06.666259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:32:06.667468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:32:06.725463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1029/2052] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:32:06.735351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:32:06.736742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:32:06.792773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11200b6a-ef8f-4441-acd2-5d638a99f2ed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:32:06.796431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:32:06.796983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:32:06.797260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:32:06.797561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:32:06.807058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:32:06.808464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:32:06.818490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1024/2053] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:32:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:32:07.222941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:32:07.224470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:32:07.240642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722d64c4-c917-4b43-86c1-c2dc5310806d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:32:07.241519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:32:07.242842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:32:07.242842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:32:07.242842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] PATH_INFO: `/v3/projects/b912c2edc2394dafbb0a9855416a6b19` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:32:07.243872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b912c2edc2394dafbb0a9855416a6b19)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:32:07.245809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:32:07.272076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:32:07.345836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b912c2edc2394dafbb0a9855416a6b19', 'request_id': 'req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.352152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b912c2edc2394dafbb0a9855416a6b19', 'request_id': 'req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.358517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b912c2edc2394dafbb0a9855416a6b19', 'request_id': 'req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.447316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] The token cache is being invalidate because project b912c2edc2394dafbb0a9855416a6b19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:32:07.447636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.448268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.449852 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1030/2054] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:32:07 2022] DELETE /identity/v3/projects/b912c2edc2394dafbb0a9855416a6b19 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:32:07.453855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:32:07.463999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9639f4e6-524d-4e87-8f1a-93dcd3d018e8 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:32:07.482451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:32:07.522335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:32:07.522973 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:32:07.523259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:32:07.523540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] PATH_INFO: `/v3/users/7914e923c7f7439ab5c2c07369a4a4b8` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:32:07.524663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7914e923c7f7439ab5c2c07369a4a4b8)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:32:07.526684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:32:07.743101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7914e923c7f7439ab5c2c07369a4a4b8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.755677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7914e923c7f7439ab5c2c07369a4a4b8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.824862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7914e923c7f7439ab5c2c07369a4a4b8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.825521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7914e923c7f7439ab5c2c07369a4a4b8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.826135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7914e923c7f7439ab5c2c07369a4a4b8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:07.838788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1025/2055] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:32:07 2022] DELETE /identity/v3/users/7914e923c7f7439ab5c2c07369a4a4b8 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:32:08.256110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:32:08.257647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:32:08.285028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfd2a3c-cd4c-4499-8152-c2721943bbd5 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:32:08.343202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:32:08.343832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:32:08.344113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:32:08.344387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] PATH_INFO: `/v3/projects/f9b02b8bcf9e478bb4c8bc71bbb9b8ad` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:32:08.345535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9b02b8bcf9e478bb4c8bc71bbb9b8ad)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:32:08.347653 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:32:08.445226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9b02b8bcf9e478bb4c8bc71bbb9b8ad', 'request_id': 'req-fd007592-0bd5-44cb-aca0-f945ace80c61'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:08.450838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9b02b8bcf9e478bb4c8bc71bbb9b8ad', 'request_id': 'req-fd007592-0bd5-44cb-aca0-f945ace80c61'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:08.457115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9b02b8bcf9e478bb4c8bc71bbb9b8ad', 'request_id': 'req-fd007592-0bd5-44cb-aca0-f945ace80c61'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:08.532529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] The token cache is being invalidate because project f9b02b8bcf9e478bb4c8bc71bbb9b8ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:32:08.532750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:08.533426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:32:08.535490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1031/2056] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:32:08 2022] DELETE /identity/v3/projects/f9b02b8bcf9e478bb4c8bc71bbb9b8ad => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:32:17.835353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=43,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:32:18.529282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=14,keystone:SELECT=66,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:33:02.274035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:33:02.275561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:33:02.346911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a211d0e1-b434-4a41-b996-4982544416e3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:33:02.406385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:33:02.407508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:33:02.407856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:33:02.408200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:33:02.410758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:33:02.442233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:33:02.443682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:33:02.457247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd007592-0bd5-44cb-aca0-f945ace80c61 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:33:02.466118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:33:02.466118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:33:02.466118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:33:02.466118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:33:02.466544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:33:02.497072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:33:02.498619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:33:02.512021 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1026/2057] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:33:02 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:33:02.558486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:33:02.559974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:33:02.563861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:33:02.565362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:33:02.570743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c3192e-10a1-4c39-90ed-490be39765f6 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:33:02.574827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:33:02.575440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:33:02.575721 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:33:02.576067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:33:02.576989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1032/2058] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:33:02 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:33:02.578059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:33:02.626218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:33:02.628913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:33:02.635278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb30bfb4-ea8e-4a42-a36e-6b2830ea5cf4 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:33:02.639009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:33:02.639614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:33:02.639897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:33:02.640167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:33:02.642089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] Could not find project: b912c2edc2394dafbb0a9855416a6b19.: keystone.exception.ProjectNotFound: Could not find project: b912c2edc2394dafbb0a9855416a6b19. Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 28 20:33:02.655387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 20:33:02.656192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 20:33:02.656973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application with Lock( Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._enter() Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.creator() Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application created_value = creator( Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 20:33:02.657768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 20:33:02.658610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: b912c2edc2394dafbb0a9855416a6b19. Oct 28 20:33:02.658610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: ERROR keystone.server.flask.application Oct 28 20:33:02.658610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1033/2059] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:33:02 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 31 msecs (HTTP/1.1 404) 5 headers in 184 bytes (2 switches on core 0) Oct 28 20:33:02.660875 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:33:02.662406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:33:02.672526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1027/2060] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:33:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:33:12.645891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=37 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:33:12.687844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=11 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:34:53.200977 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:34:53.203020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:34:53.328507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fc0d6-dd1b-4ede-b5d0-99c9d899bcfb service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:34:53.333573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:34:53.334175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:34:53.334454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:34:53.334726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:34:53.336439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:34:53.348367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:34:53.349986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:34:53.360873 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1034/2061] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:34:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:02.879157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:02.888689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e07015-d116-4894-ad81-4b6a4f424323 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:02.892367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:02.892974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:02.893258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:02.893590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:02.895291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] Could not find project: fac1485ccdeb434ca49caa52ebfdc24b.: keystone.exception.ProjectNotFound: Could not find project: fac1485ccdeb434ca49caa52ebfdc24b. Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 28 20:35:02.908540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 20:35:02.914927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 20:35:02.916562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 20:35:02.917718 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 20:35:02.918823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: fac1485ccdeb434ca49caa52ebfdc24b. Oct 28 20:35:02.918823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 20:35:02.918823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1028/2062] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:02 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 31 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Oct 28 20:35:03.356773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=54 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:35:04.123617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:04.126893 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:04.327149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124da984-cb58-4203-8884-53713cbd26c6 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:04.333160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e48bc-c310-4203-a968-c62e47722ae3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:04.333841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e48bc-c310-4203-a968-c62e47722ae3 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:04.334273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e48bc-c310-4203-a968-c62e47722ae3 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:04.334561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e48bc-c310-4203-a968-c62e47722ae3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:04.336410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3f4e48bc-c310-4203-a968-c62e47722ae3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:04.365849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e48bc-c310-4203-a968-c62e47722ae3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:04.378197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e48bc-c310-4203-a968-c62e47722ae3 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:04.406264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1035/2063] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:04.490624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:04.507503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5433370d-286a-4ff5-a446-86606d7f03c5 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:04.513113 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:04.515261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:04.515746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:04.516280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:04.518370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:04.532646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:04.534619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:04.568102 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1029/2064] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:07.151556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:07.152435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:07.152663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:07.181047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] MFA Rules not processed for user `15db5106efbd434f99a4d102810cc7a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:35:07.182295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:07.237451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1036/2065] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:35:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:35:07.589001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:07.590478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:07.654043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd14f31-be5e-41c5-9c6f-0138cacbec90 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:07.657675 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:07.658297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:07.658585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:07.658913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:07.660635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:07.670136 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:07.671558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:07.679339 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1030/2066] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:08.130466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:08.131954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:08.149470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8e5c05-ec23-48fa-9c0d-9eec57113555 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:08.156176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:08.156847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:08.157303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:08.157611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:08.159942 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:08.172776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:08.174421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:08.196137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1037/2067] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:09.817586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:09.819521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:09.830570 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52a0ea1-c408-4a4c-b3bb-ab1e13a9705e service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:09.834880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:09.835497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:09.835788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:09.836136 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:09.837933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:09.904914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:09.906421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:09.916342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1031/2068] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:13.498606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:13.509899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05954c4d-cf91-433a-87a6-73ec1f0ec75a service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:13.515304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:13.516078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:13.516519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:13.516801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:13.518621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:13.531773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:13.533462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:13.545626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1038/2069] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:13.605077 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:13.607522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:13.620240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086520a-48e3-4fe1-81c5-400a86843622 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:13.625669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:13.626382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:13.626665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:13.626966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] PATH_INFO: `/v3/users/ce7f52b7f96c400fb7d5fb0453c94362` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:13.628085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce7f52b7f96c400fb7d5fb0453c94362)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:13.630045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:13.817958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce7f52b7f96c400fb7d5fb0453c94362'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:13.831979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce7f52b7f96c400fb7d5fb0453c94362'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:13.835823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:35:13.921606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce7f52b7f96c400fb7d5fb0453c94362'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:13.922280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce7f52b7f96c400fb7d5fb0453c94362'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:13.922896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0ce0ce97-e6e7-4f0e-9a87-273ca6393a23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce7f52b7f96c400fb7d5fb0453c94362'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:13.936836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1032/2070] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:35:13 2022] DELETE /identity/v3/users/ce7f52b7f96c400fb7d5fb0453c94362 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:35:14.209263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:14.222341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad1080d-749a-4c98-9766-3a3bba2b9790 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:14.291480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:35:14.297713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:14.304498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.304498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.304498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] PATH_INFO: `/v3/projects/ab907bf956554681beae45b8973db6e5` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.304498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab907bf956554681beae45b8973db6e5)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:14.304498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:14.412908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab907bf956554681beae45b8973db6e5', 'request_id': 'req-73a73a07-b8fc-4c11-a17a-c5fff9790053'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:14.418812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab907bf956554681beae45b8973db6e5', 'request_id': 'req-73a73a07-b8fc-4c11-a17a-c5fff9790053'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:14.425846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab907bf956554681beae45b8973db6e5', 'request_id': 'req-73a73a07-b8fc-4c11-a17a-c5fff9790053'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:14.507424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] The token cache is being invalidate because project ab907bf956554681beae45b8973db6e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:35:14.507974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:14.508392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73a73a07-b8fc-4c11-a17a-c5fff9790053 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:35:14.509917 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1039/2071] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:35:14 2022] DELETE /identity/v3/projects/ab907bf956554681beae45b8973db6e5 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:35:14.516650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79352f59-d5e5-4aa5-a2be-ea3f2cf09535 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.520472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79352f59-d5e5-4aa5-a2be-ea3f2cf09535 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.520472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79352f59-d5e5-4aa5-a2be-ea3f2cf09535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.538559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-79352f59-d5e5-4aa5-a2be-ea3f2cf09535 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:35:14.539757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-79352f59-d5e5-4aa5-a2be-ea3f2cf09535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:14.600429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1033/2072] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:35:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:35:14.608540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d78eec-2f0f-472c-975c-9d4987807095 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.608830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d78eec-2f0f-472c-975c-9d4987807095 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.609108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d78eec-2f0f-472c-975c-9d4987807095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.629273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e1d78eec-2f0f-472c-975c-9d4987807095 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:35:14.630444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e1d78eec-2f0f-472c-975c-9d4987807095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:14.638918 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1040/2073] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:35:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:35:14.643145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79352f59-d5e5-4aa5-a2be-ea3f2cf09535 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:14.653520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79352f59-d5e5-4aa5-a2be-ea3f2cf09535 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:14.658345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:14.658953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.659241 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.659523 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.660173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:35:14.660482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:14.662132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:14.673608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1034/2074] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:35:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:35:14.677860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d78eec-2f0f-472c-975c-9d4987807095 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:14.687109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d78eec-2f0f-472c-975c-9d4987807095 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:14.691663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:14.692285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.692972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.692972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.693511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:14.695353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:14.791680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1041/2075] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:35:14 2022] POST /identity/v3/projects => generated 371 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:35:14.795841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:14.805753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ecd7c3-3ef7-4ec7-b018-33795ac313bc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:14.861969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:14.862579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.862863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.863141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.863795 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:14.865835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:14.933391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1035/2076] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:35:14 2022] POST /identity/v3/users => generated 286 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:35:14.937601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:14.948048 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cca7547-4ea9-4494-bbe6-a79a6661bd1b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:14.953266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:14.953896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.956098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.956098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.956098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:14.956680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:14.966623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1042/2077] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:35:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:35:14.970573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:14.980477 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad92a66-879f-4620-8309-32e4e937ca2a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:14.986049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db0b085-9264-4d44-ba43-105fd190637c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:14.986641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0b085-9264-4d44-ba43-105fd190637c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:14.986922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0b085-9264-4d44-ba43-105fd190637c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:14.987197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0b085-9264-4d44-ba43-105fd190637c admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:14.987829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db0b085-9264-4d44-ba43-105fd190637c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:14.989438 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db0b085-9264-4d44-ba43-105fd190637c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:15.001306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1036/2078] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:35:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:35:15.005149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:15.015136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0febd93f-d56f-4bd7-80d1-fe01d4765cb9 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:15.019682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:15.020328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:15.020606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:15.020877 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] PATH_INFO: `/v3/projects/9a278c3aacea4192a9c38fb541300238/users/accf2a5b89fa4623b95a968ee0383eb4/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:15.037683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a278c3aacea4192a9c38fb541300238, user_id=accf2a5b89fa4623b95a968ee0383eb4, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:15.040346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:15.104719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1043/2079] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:35:15 2022] PUT /identity/v3/projects/9a278c3aacea4192a9c38fb541300238/users/accf2a5b89fa4623b95a968ee0383eb4/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:35:15.115887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:15.116205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:15.116486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:15.144547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] MFA Rules not processed for user `accf2a5b89fa4623b95a968ee0383eb4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:35:15.146761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:15.190337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:35:15.212966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1037/2080] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:35:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:35:15.225027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:15.226755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:15.304036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e986a8a-e4f9-4e1e-9ab7-1aa8bcec2f15 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:15.309991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:15.310604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:15.313025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:15.313025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:15.319684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:15.321314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:15.331513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1044/2081] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:15 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:18.416101 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:35:18.417609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:18.433619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d9e0f1-14ed-4203-9b29-e64d2e092c5b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:35:18.439216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82587c5-9c20-4560-baf5-39e49e85b9b4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:35:18.439830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82587c5-9c20-4560-baf5-39e49e85b9b4 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:35:18.440116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82587c5-9c20-4560-baf5-39e49e85b9b4 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:35:18.440396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82587c5-9c20-4560-baf5-39e49e85b9b4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:35:18.442437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a82587c5-9c20-4560-baf5-39e49e85b9b4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:35:18.454694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82587c5-9c20-4560-baf5-39e49e85b9b4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:35:18.456297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82587c5-9c20-4560-baf5-39e49e85b9b4 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:35:18.467385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1038/2082] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:35:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:35:25.327117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:35:28.462296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=14 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:36:15.287071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:15.351399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa95848-70d6-4b5f-b7cd-7c5a62140288 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:15.355957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:15.356579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:15.356860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:15.357923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] PATH_INFO: `/v3/users/accf2a5b89fa4623b95a968ee0383eb4` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:15.359041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] RBAC: Authorizing `identity:delete_user(user_id=accf2a5b89fa4623b95a968ee0383eb4)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:15.360963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:15.570071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'accf2a5b89fa4623b95a968ee0383eb4'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:15.570668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'accf2a5b89fa4623b95a968ee0383eb4'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:15.583230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'accf2a5b89fa4623b95a968ee0383eb4'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:15.590374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'accf2a5b89fa4623b95a968ee0383eb4'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:15.591095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'accf2a5b89fa4623b95a968ee0383eb4'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:15.663768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1045/2083] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:36:15 2022] DELETE /identity/v3/users/accf2a5b89fa4623b95a968ee0383eb4 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:36:15.668247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:15.668741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:15.668835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:15.689006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:36:15.690246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:15.748956 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1039/2084] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:36:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:36:15.758577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:15.759982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:15.844467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f576ad31-03d4-49d7-bdac-9237e7e87697 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:15.848138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d1e483-1798-48b1-9cbf-4926129fc381 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:15.848744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d1e483-1798-48b1-9cbf-4926129fc381 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:15.849029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d1e483-1798-48b1-9cbf-4926129fc381 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:15.849303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d1e483-1798-48b1-9cbf-4926129fc381 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:15.855143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:36:15.859668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d1e483-1798-48b1-9cbf-4926129fc381 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:15.861131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d1e483-1798-48b1-9cbf-4926129fc381 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:15.870621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1046/2085] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:36:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:36:16.281388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:16.283252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:16.294914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dbdc0a-f707-4f18-933c-0918baeb123d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:16.299843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:16.301452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.302085 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.302374 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] PATH_INFO: `/v3/projects/9a278c3aacea4192a9c38fb541300238` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.303589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a278c3aacea4192a9c38fb541300238)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:16.306768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:16.408379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a278c3aacea4192a9c38fb541300238', 'request_id': 'req-2ca95c61-add0-4ede-8e3c-197a708f12ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:16.452854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a278c3aacea4192a9c38fb541300238', 'request_id': 'req-2ca95c61-add0-4ede-8e3c-197a708f12ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:16.459394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a278c3aacea4192a9c38fb541300238', 'request_id': 'req-2ca95c61-add0-4ede-8e3c-197a708f12ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:16.470453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] The token cache is being invalidate because project 9a278c3aacea4192a9c38fb541300238 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:36:16.476017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:16.476017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ca95c61-add0-4ede-8e3c-197a708f12ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:36:16.476017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1040/2086] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:36:16 2022] DELETE /identity/v3/projects/9a278c3aacea4192a9c38fb541300238 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:36:16.477780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d9c769-33af-47f0-9685-2bfea567624e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.478230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d9c769-33af-47f0-9685-2bfea567624e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.478360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d9c769-33af-47f0-9685-2bfea567624e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.499487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-91d9c769-33af-47f0-9685-2bfea567624e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:36:16.500786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-91d9c769-33af-47f0-9685-2bfea567624e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:16.558575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1047/2087] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:36:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:36:16.568081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489620c9-29d0-4526-a89f-1f13c01b14a5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.568370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489620c9-29d0-4526-a89f-1f13c01b14a5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.568642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489620c9-29d0-4526-a89f-1f13c01b14a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.588475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-489620c9-29d0-4526-a89f-1f13c01b14a5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:36:16.589694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-489620c9-29d0-4526-a89f-1f13c01b14a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:16.597573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1041/2088] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:36:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:36:16.601822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d9c769-33af-47f0-9685-2bfea567624e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:16.611725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d9c769-33af-47f0-9685-2bfea567624e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:16.616357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:16.616959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.617238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.617540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.618182 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:36:16.618478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:16.620071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:16.632250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1048/2089] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:36:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:36:16.636349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489620c9-29d0-4526-a89f-1f13c01b14a5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:16.646071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489620c9-29d0-4526-a89f-1f13c01b14a5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:16.650671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:16.651262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.651542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.651816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.652432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:16.654390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:16.734391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1042/2090] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:36:16 2022] POST /identity/v3/projects => generated 373 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:36:16.738860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:16.750122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb475c15-e4fd-4d27-b6d8-d5d22836dc7c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:16.806324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:16.806936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.807218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.807493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.808124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:16.810706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:16.884988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1049/2091] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:36:16 2022] POST /identity/v3/users => generated 287 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:36:16.889120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:16.899383 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006f96f-b225-4b43-8648-37f52c5fcf19 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:16.904672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:16.905280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.905598 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.905876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.906519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:16.908158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:16.918008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1043/2092] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:36:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:36:16.922631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:16.932775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cf982-58a4-4940-9992-a21978cb4ef2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:16.938603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da250ae7-2d0f-4a6e-900b-55d41510e917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:16.939200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da250ae7-2d0f-4a6e-900b-55d41510e917 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.939485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da250ae7-2d0f-4a6e-900b-55d41510e917 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.939763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da250ae7-2d0f-4a6e-900b-55d41510e917 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.940401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da250ae7-2d0f-4a6e-900b-55d41510e917 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:16.942037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da250ae7-2d0f-4a6e-900b-55d41510e917 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:16.953133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1050/2093] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:36:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:36:16.957138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:16.967087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00342940-8b5d-4513-8b30-64ebbcf6c65b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:16.971662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:16.972306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:16.972588 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:16.972862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] PATH_INFO: `/v3/projects/e77ed36e4a894aa89c375a13af93a71b/users/ce72c24b489a4d929545e400d73f94ce/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:16.988281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e77ed36e4a894aa89c375a13af93a71b, user_id=ce72c24b489a4d929545e400d73f94ce, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:16.990929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:17.068139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1044/2094] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:36:16 2022] PUT /identity/v3/projects/e77ed36e4a894aa89c375a13af93a71b/users/ce72c24b489a4d929545e400d73f94ce/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:36:17.078115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bd6e0d-3fb0-493d-95e5-255cfd6f8a98 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.078403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bd6e0d-3fb0-493d-95e5-255cfd6f8a98 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.078679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bd6e0d-3fb0-493d-95e5-255cfd6f8a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.105602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-76bd6e0d-3fb0-493d-95e5-255cfd6f8a98 None None] MFA Rules not processed for user `ce72c24b489a4d929545e400d73f94ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:36:17.106790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-76bd6e0d-3fb0-493d-95e5-255cfd6f8a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:17.154340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:36:17.171792 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1051/2095] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:36:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:36:17.176365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:17.198135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:36:17.239389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49aa28b-791a-4f48-81e3-2aead9a25900 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:17.243967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:17.244567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.244850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.245122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.245781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:17.247797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:17.345203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1045/2096] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:36:17 2022] POST /identity/v3/projects => generated 377 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:36:17.348568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bd6e0d-3fb0-493d-95e5-255cfd6f8a98 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:17.360278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bd6e0d-3fb0-493d-95e5-255cfd6f8a98 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:17.422061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:17.422678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.422961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.423242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.423889 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:17.425897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:17.572719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1052/2097] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:36:17 2022] POST /identity/v3/users => generated 295 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:36:17.577662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:17.588066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7a798-e038-4eba-ad6f-65f7fa5f13a6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:17.593685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:17.594293 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.594581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.594862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.595502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:17.597110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:17.608416 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1046/2098] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:36:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:36:17.613387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:17.623425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ea22b2-e4a2-4856-8e3d-7d43b926ba24 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:17.628009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:17.628685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.628970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.629243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] PATH_INFO: `/v3/projects/21165136cdae4683b5f427a57e53ea97/users/2b8a548ef8e044758fe2d28702bf08bb/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.645455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21165136cdae4683b5f427a57e53ea97, user_id=2b8a548ef8e044758fe2d28702bf08bb, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:17.648086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:17.715610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1053/2099] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:36:17 2022] PUT /identity/v3/projects/21165136cdae4683b5f427a57e53ea97/users/2b8a548ef8e044758fe2d28702bf08bb/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:36:17.720240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:17.730110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9ce797-98f9-4c64-9d39-37c423b15e02 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:17.785678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22f5ae8-92a5-4303-bfb5-64e31e1f263d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:17.786299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22f5ae8-92a5-4303-bfb5-64e31e1f263d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.786573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22f5ae8-92a5-4303-bfb5-64e31e1f263d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.786844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22f5ae8-92a5-4303-bfb5-64e31e1f263d admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.787478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22f5ae8-92a5-4303-bfb5-64e31e1f263d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:17.789054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22f5ae8-92a5-4303-bfb5-64e31e1f263d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:17.799437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1047/2100] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:36:17 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:36:17.803566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:17.814069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897c5d56-c673-4fea-bfce-146b4ee953d2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:17.818623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:17.819298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.819577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.819848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] PATH_INFO: `/v3/projects/21165136cdae4683b5f427a57e53ea97/users/2b8a548ef8e044758fe2d28702bf08bb/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.821933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21165136cdae4683b5f427a57e53ea97, user_id=2b8a548ef8e044758fe2d28702bf08bb, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:17.824683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:17.835701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1054/2101] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:36:17 2022] PUT /identity/v3/projects/21165136cdae4683b5f427a57e53ea97/users/2b8a548ef8e044758fe2d28702bf08bb/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:36:17.846137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:17.846430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:17.846711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:17.874370 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] MFA Rules not processed for user `2b8a548ef8e044758fe2d28702bf08bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:36:17.875629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:17.943509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1048/2102] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:36:17 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:36:27.831913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=49,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:36:27.940575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:36:57.047496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:57.049192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:57.106097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb77a088-0baa-45e9-bdd3-8ffd36568838 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:57.109912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:57.110513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:57.110793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:57.111066 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:57.112767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:57.187530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:57.188960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:57.209398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1055/2103] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:36:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:36:58.716809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:36:58.718362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:58.730723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5dc55e-5b38-40cb-bdf7-ec1185762fe6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:36:58.734490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:36:58.735096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:36:58.735378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:36:58.735650 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:36:58.738024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:36:58.747618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:36:58.749092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:36:58.758183 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1049/2104] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:36:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:37:00.718059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:00.719478 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:00.733777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168d5f17-b7a6-455f-8bab-05949292bc66 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:00.739858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:00.740459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:00.740744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:00.741021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:00.742727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:00.810983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:00.812587 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:00.831811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1056/2105] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:37:00 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:37:03.772453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:03.773868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:03.839112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621ce122-cdcc-46a3-bc0d-6167ee5aec50 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:03.842763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:03.843366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:03.843649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:03.843943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:03.845635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:03.904539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:03.906002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:03.915499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1050/2106] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:37:03 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:37:04.009832 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:04.019178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81ed005-25e8-45cd-802a-c0d33461cd0b service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:04.022802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:04.023401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:04.023683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:04.023958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:04.025706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:04.036055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:37:04.039962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:04.041509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:04.052951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1057/2107] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:37:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:37:04.828928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:04.830536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:04.841864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621115ef-d56f-4721-92b2-fec1a49727ae service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:04.845663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:04.846363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:04.846741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:04.847083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:04.907240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:04.908723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:04.918227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1051/2108] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:37:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:37:05.290706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:05.300257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208d6e36-f197-4c50-b835-a45324269e90 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:05.304654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:05.305262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:05.305605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:05.305886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:05.307671 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:05.373192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:05.374683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:05.385631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1058/2109] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:37:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:37:09.379752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:09.381472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:09.423330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:37:09.458524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1deec67-9d1c-483b-b6f4-90060dd68a5c service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:09.463696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:09.464306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:09.464591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:09.464869 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:09.466660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:09.478927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:09.480145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:09.490995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1052/2110] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:37:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:37:09.552487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:09.554906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:09.569430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda130b3-da5a-415a-a9cf-ab4e292a9f3a service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:09.573518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:09.574088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:09.574372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:09.574648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] PATH_INFO: `/v3/users/e0ba6bf2459d4a98a3a1f5684bdad2b2` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:09.575762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0ba6bf2459d4a98a3a1f5684bdad2b2)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:09.577717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:09.773538 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0ba6bf2459d4a98a3a1f5684bdad2b2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:09.774193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0ba6bf2459d4a98a3a1f5684bdad2b2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:09.786404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0ba6bf2459d4a98a3a1f5684bdad2b2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:09.793210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0ba6bf2459d4a98a3a1f5684bdad2b2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:09.793980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-946c3c46-a43e-4442-a158-731cec3cfdbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0ba6bf2459d4a98a3a1f5684bdad2b2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:09.882844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1059/2111] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:37:09 2022] DELETE /identity/v3/users/e0ba6bf2459d4a98a3a1f5684bdad2b2 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:37:10.122944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:10.134174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf91101-fd78-4296-8be4-8678304f7adc service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:10.191242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:10.192703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.193003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.193285 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] PATH_INFO: `/v3/projects/34d30fbe6fa645bd9ea3d88ea5f68449` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.203780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=34d30fbe6fa645bd9ea3d88ea5f68449)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:10.205757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:10.309133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34d30fbe6fa645bd9ea3d88ea5f68449', 'request_id': 'req-ce218382-f089-4673-82be-b0b59f5c25cf'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:10.370746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34d30fbe6fa645bd9ea3d88ea5f68449', 'request_id': 'req-ce218382-f089-4673-82be-b0b59f5c25cf'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:10.377749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34d30fbe6fa645bd9ea3d88ea5f68449', 'request_id': 'req-ce218382-f089-4673-82be-b0b59f5c25cf'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:10.389267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] The token cache is being invalidate because project 34d30fbe6fa645bd9ea3d88ea5f68449 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:37:10.389851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:10.390261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ce218382-f089-4673-82be-b0b59f5c25cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:37:10.391738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1053/2112] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:37:10 2022] DELETE /identity/v3/projects/34d30fbe6fa645bd9ea3d88ea5f68449 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:37:10.397795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a67b6d-2fa5-40ec-a5e1-181793acc672 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.398235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a67b6d-2fa5-40ec-a5e1-181793acc672 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.398540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a67b6d-2fa5-40ec-a5e1-181793acc672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.418827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-49a67b6d-2fa5-40ec-a5e1-181793acc672 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:37:10.420471 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-49a67b6d-2fa5-40ec-a5e1-181793acc672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:10.473940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:37:10.482326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1060/2113] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:37:10.491245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523ee782-48b3-4922-801c-716ca7cae5f0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.491534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523ee782-48b3-4922-801c-716ca7cae5f0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.491815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523ee782-48b3-4922-801c-716ca7cae5f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.511495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-523ee782-48b3-4922-801c-716ca7cae5f0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:37:10.512695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-523ee782-48b3-4922-801c-716ca7cae5f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:10.521175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1054/2114] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:37:10.525695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a67b6d-2fa5-40ec-a5e1-181793acc672 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:10.535812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a67b6d-2fa5-40ec-a5e1-181793acc672 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:10.540421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:10.541016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.541298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.541616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.542239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:37:10.542543 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:10.544131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:10.556088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1061/2115] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:37:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:37:10.560333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523ee782-48b3-4922-801c-716ca7cae5f0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:10.569701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523ee782-48b3-4922-801c-716ca7cae5f0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:10.574382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:10.574977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.575262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.575537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.576167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:10.578118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:10.677056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1055/2116] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:37:10 2022] POST /identity/v3/projects => generated 365 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:37:10.681191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:10.691129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e6b606-facd-4811-9c14-ebe4c2557f9e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:10.749274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:10.749922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.750211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.750489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.751116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:10.753046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:10.819540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1062/2117] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:37:10 2022] POST /identity/v3/users => generated 283 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:37:10.823699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:10.835237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a071bc-e079-406b-93e4-bbf2b419086c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:10.839856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:10.840458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.840740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.841015 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.841699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:10.843289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:10.853234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1056/2118] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:37:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:37:10.857183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:10.867720 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4677a013-e24a-4954-b7c0-8e565b228702 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:10.872472 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459f17f7-8fb4-4cfa-a010-5262f0aca0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:10.873072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459f17f7-8fb4-4cfa-a010-5262f0aca0be admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.873391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459f17f7-8fb4-4cfa-a010-5262f0aca0be admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.873677 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459f17f7-8fb4-4cfa-a010-5262f0aca0be admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.874319 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459f17f7-8fb4-4cfa-a010-5262f0aca0be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:10.875923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459f17f7-8fb4-4cfa-a010-5262f0aca0be admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:10.888291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1063/2119] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:37:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:37:10.892218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:37:10.898504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:37:10.904170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78533d1a-a8ea-4c88-8aea-ba13d468887b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:37:10.909425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8da67e6-261c-4a96-be01-b52d1e996c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:37:10.910126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8da67e6-261c-4a96-be01-b52d1e996c87 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:10.910453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8da67e6-261c-4a96-be01-b52d1e996c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:10.910799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8da67e6-261c-4a96-be01-b52d1e996c87 admin admin] PATH_INFO: `/v3/projects/91a580f071644e258dca333900aa83b2/users/182cd571ea974bcfa752d93a918bde68/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:10.925729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8da67e6-261c-4a96-be01-b52d1e996c87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91a580f071644e258dca333900aa83b2, user_id=182cd571ea974bcfa752d93a918bde68, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:37:10.928437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8da67e6-261c-4a96-be01-b52d1e996c87 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:37:10.999582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1057/2120] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:37:10 2022] PUT /identity/v3/projects/91a580f071644e258dca333900aa83b2/users/182cd571ea974bcfa752d93a918bde68/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:37:11.067922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6336ad4f-2d3c-4757-abcc-c1e9c0948718 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:11.068216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6336ad4f-2d3c-4757-abcc-c1e9c0948718 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:11.068497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6336ad4f-2d3c-4757-abcc-c1e9c0948718 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:11.096356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6336ad4f-2d3c-4757-abcc-c1e9c0948718 None None] MFA Rules not processed for user `182cd571ea974bcfa752d93a918bde68`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:37:11.097541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6336ad4f-2d3c-4757-abcc-c1e9c0948718 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:11.165786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1064/2121] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:37:11 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:37:12.908891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78022ddb-6963-4226-86bc-3ae7ebb6245c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:37:12.909408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78022ddb-6963-4226-86bc-3ae7ebb6245c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:37:12.909502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78022ddb-6963-4226-86bc-3ae7ebb6245c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:37:12.929027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-78022ddb-6963-4226-86bc-3ae7ebb6245c None None] MFA Rules not processed for user `724baaa4aa834e609fe4f513f9fc0b4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:37:12.930245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-78022ddb-6963-4226-86bc-3ae7ebb6245c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:37:12.983234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1058/2122] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:37:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:37:21.163588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=72,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:37:22.976610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=34,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:38:11.228609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6336ad4f-2d3c-4757-abcc-c1e9c0948718 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:11.292345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6336ad4f-2d3c-4757-abcc-c1e9c0948718 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:11.297052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:11.297711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:11.298000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:11.298280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] PATH_INFO: `/v3/users/182cd571ea974bcfa752d93a918bde68` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:11.299391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] RBAC: Authorizing `identity:delete_user(user_id=182cd571ea974bcfa752d93a918bde68)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:11.301475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:11.481496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '182cd571ea974bcfa752d93a918bde68'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:11.482152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '182cd571ea974bcfa752d93a918bde68'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:11.494678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '182cd571ea974bcfa752d93a918bde68'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:11.502283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '182cd571ea974bcfa752d93a918bde68'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:11.503002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '182cd571ea974bcfa752d93a918bde68'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:11.566018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1065/2123] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:38:11 2022] DELETE /identity/v3/users/182cd571ea974bcfa752d93a918bde68 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:38:11.570455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:11.570945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:11.571043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:11.591026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:38:11.599229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:11.652194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1059/2124] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:38:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:38:11.662346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:11.663550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:11.732396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbd05f7-b13c-4991-ac27-1271ae874470 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:11.737159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690ed652-7b3b-4707-9a99-a29d94054fd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:11.737751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ed652-7b3b-4707-9a99-a29d94054fd0 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:11.738032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ed652-7b3b-4707-9a99-a29d94054fd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:11.738304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ed652-7b3b-4707-9a99-a29d94054fd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:11.747278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690ed652-7b3b-4707-9a99-a29d94054fd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:11.749398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690ed652-7b3b-4707-9a99-a29d94054fd0 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:11.758960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1066/2125] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:38:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:38:12.174695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:12.176253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:12.190648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ccdde5-dc49-4ec1-a4ef-98a9df8ee785 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:12.195398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:12.196040 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.196323 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.196600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] PATH_INFO: `/v3/projects/91a580f071644e258dca333900aa83b2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.197794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91a580f071644e258dca333900aa83b2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:12.199895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:12.333975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91a580f071644e258dca333900aa83b2', 'request_id': 'req-d959b138-8565-4049-9033-87eee7f4ef33'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:12.406964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91a580f071644e258dca333900aa83b2', 'request_id': 'req-d959b138-8565-4049-9033-87eee7f4ef33'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:12.413733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91a580f071644e258dca333900aa83b2', 'request_id': 'req-d959b138-8565-4049-9033-87eee7f4ef33'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:12.426072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] The token cache is being invalidate because project 91a580f071644e258dca333900aa83b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:38:12.426298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:12.426932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d959b138-8565-4049-9033-87eee7f4ef33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:38:12.428475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1060/2126] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:38:12 2022] DELETE /identity/v3/projects/91a580f071644e258dca333900aa83b2 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:38:12.443377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771f95a4-e87b-4613-a685-4794ad4f236b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.443872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771f95a4-e87b-4613-a685-4794ad4f236b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.443967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771f95a4-e87b-4613-a685-4794ad4f236b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.464043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-771f95a4-e87b-4613-a685-4794ad4f236b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:38:12.465234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-771f95a4-e87b-4613-a685-4794ad4f236b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:12.473005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:38:12.526528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1067/2127] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:38:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:38:12.535466 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43748672-8f5a-44ef-a88a-0a6570184291 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.535759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43748672-8f5a-44ef-a88a-0a6570184291 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.536039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43748672-8f5a-44ef-a88a-0a6570184291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.555671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-43748672-8f5a-44ef-a88a-0a6570184291 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:38:12.556864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-43748672-8f5a-44ef-a88a-0a6570184291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:12.565145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1061/2128] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:38:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:38:12.569360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771f95a4-e87b-4613-a685-4794ad4f236b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:12.579484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771f95a4-e87b-4613-a685-4794ad4f236b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:12.584148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:12.584744 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.585028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.585302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.585982 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:38:12.586286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:12.587876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:12.599936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1068/2129] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:38:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:38:12.604310 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43748672-8f5a-44ef-a88a-0a6570184291 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:12.614440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43748672-8f5a-44ef-a88a-0a6570184291 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:12.619142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:12.619741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.620027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.620352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.620970 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:12.622909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:12.728347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1062/2130] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:38:12 2022] POST /identity/v3/projects => generated 363 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:38:12.732598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:12.743954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f236ac2-2f5a-4c0a-9316-727a02592685 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:12.801644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:12.802248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.802532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.802812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.803441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:12.805391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:12.886809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1069/2131] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:38:12 2022] POST /identity/v3/users => generated 282 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:38:12.891100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:12.901840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93bfde5-e17a-43cb-a24d-6476e1aa9c16 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:12.907528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:12.908132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.908462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.908737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.909452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:12.911065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:12.922070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1063/2132] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:38:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:38:12.926177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:12.936127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a080b89e-4716-4e2a-b222-c23eeb182c6d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:12.940776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ba7699-5c36-48d9-a4ee-11cef795216f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:12.941408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba7699-5c36-48d9-a4ee-11cef795216f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.941695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba7699-5c36-48d9-a4ee-11cef795216f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.941968 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba7699-5c36-48d9-a4ee-11cef795216f admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.942600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ba7699-5c36-48d9-a4ee-11cef795216f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:12.944189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ba7699-5c36-48d9-a4ee-11cef795216f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:12.954945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1070/2133] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:38:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:38:12.959026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:12.970054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cde57d-68b9-4682-97c5-a2ae35291105 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:12.974812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:12.975469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:12.975753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:12.976034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] PATH_INFO: `/v3/projects/73502baaf3ba4f26894cc22e3978dc59/users/8c2e59073d7b44db8d32417e742c489f/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:12.992478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73502baaf3ba4f26894cc22e3978dc59, user_id=8c2e59073d7b44db8d32417e742c489f, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:12.995095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:13.094107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1064/2134] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:38:12 2022] PUT /identity/v3/projects/73502baaf3ba4f26894cc22e3978dc59/users/8c2e59073d7b44db8d32417e742c489f/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:38:13.104603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e35f4a-ae46-42ce-9e30-2cb4db4a603d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.104885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e35f4a-ae46-42ce-9e30-2cb4db4a603d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.105176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e35f4a-ae46-42ce-9e30-2cb4db4a603d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.132787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-15e35f4a-ae46-42ce-9e30-2cb4db4a603d None None] MFA Rules not processed for user `8c2e59073d7b44db8d32417e742c489f`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:38:13.134004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-15e35f4a-ae46-42ce-9e30-2cb4db4a603d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:13.196137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1071/2135] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:38:13 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:38:13.200909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:13.223047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:38:13.263520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdd73ae-f051-4dd9-882d-58253e406147 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:13.268803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:13.269438 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.269733 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.270009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.270696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:13.272611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:13.334373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1065/2136] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:38:13 2022] POST /identity/v3/projects => generated 363 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:38:13.338645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e35f4a-ae46-42ce-9e30-2cb4db4a603d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:13.350211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e35f4a-ae46-42ce-9e30-2cb4db4a603d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:13.353757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:38:13.409471 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:13.410121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.410406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.410687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.411362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:13.413275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:13.537041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1072/2137] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:38:13 2022] POST /identity/v3/users => generated 288 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:38:13.541380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:13.551505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794298a1-8250-4e3e-93c0-9755803585d8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:13.556319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:13.556936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.557252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.557564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.558210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:13.559813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:13.571266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1066/2138] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:38:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:38:13.575304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:13.586146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0f4da7-a137-48f5-8f3d-a31071aad7fa admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:13.590993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:13.591635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.591919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.592197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] PATH_INFO: `/v3/projects/3b0f76eba64d4597b254c0cfb1f504f0/users/7a51768350e14f51947042a65af1df74/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.606821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b0f76eba64d4597b254c0cfb1f504f0, user_id=7a51768350e14f51947042a65af1df74, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:13.610118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:13.687897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1073/2139] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:38:13 2022] PUT /identity/v3/projects/3b0f76eba64d4597b254c0cfb1f504f0/users/7a51768350e14f51947042a65af1df74/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:38:13.692005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:13.702960 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f73eab-df22-407f-8975-e70af415ae5a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:13.758437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe4228d-2cc4-442a-a352-8c53b683f842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:13.759043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe4228d-2cc4-442a-a352-8c53b683f842 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.759316 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe4228d-2cc4-442a-a352-8c53b683f842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.759590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe4228d-2cc4-442a-a352-8c53b683f842 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.760224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe4228d-2cc4-442a-a352-8c53b683f842 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:13.761839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe4228d-2cc4-442a-a352-8c53b683f842 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:13.772213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1067/2140] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:38:13 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:38:13.776379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:13.786908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76956fdb-6baf-4d02-a5a6-29344a8d55d6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:13.791685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:13.792269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.792549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.792824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] PATH_INFO: `/v3/projects/3b0f76eba64d4597b254c0cfb1f504f0/users/7a51768350e14f51947042a65af1df74/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.794747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b0f76eba64d4597b254c0cfb1f504f0, user_id=7a51768350e14f51947042a65af1df74, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:13.798103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:13.808100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1074/2141] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:38:13 2022] PUT /identity/v3/projects/3b0f76eba64d4597b254c0cfb1f504f0/users/7a51768350e14f51947042a65af1df74/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:38:13.820176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:13.820468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:13.820747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:13.850747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] MFA Rules not processed for user `7a51768350e14f51947042a65af1df74`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:38:13.852056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:13.922453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1068/2142] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:38:13 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:38:23.804485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=41,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:38:23.918915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:38:59.447817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:59.449600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:59.511223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758b8c96-52c8-4444-bd50-564ad4992224 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:59.515026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7160518f-6f4b-4cd0-9a7f-99f13d5b1606 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:59.515692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7160518f-6f4b-4cd0-9a7f-99f13d5b1606 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:59.515976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7160518f-6f4b-4cd0-9a7f-99f13d5b1606 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:59.516247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7160518f-6f4b-4cd0-9a7f-99f13d5b1606 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:59.518016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7160518f-6f4b-4cd0-9a7f-99f13d5b1606 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:59.545217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:59.546898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:59.559317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2f9c68-2fff-46ab-8e80-d9c6b924b727 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:59.563077 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:59.563685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:59.563969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:59.564246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:59.566020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] Could not find project: 91a580f071644e258dca333900aa83b2.: keystone.exception.ProjectNotFound: Could not find project: 91a580f071644e258dca333900aa83b2. Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 28 20:38:59.578389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 20:38:59.579676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 20:38:59.580602 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 20:38:59.582132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 20:38:59.583109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: 91a580f071644e258dca333900aa83b2. Oct 28 20:38:59.583109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Oct 28 20:38:59.583109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1069/2143] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:38:59 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 35 msecs (HTTP/1.1 404) 5 headers in 184 bytes (2 switches on core 0) Oct 28 20:38:59.593000 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7160518f-6f4b-4cd0-9a7f-99f13d5b1606 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:59.596644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7160518f-6f4b-4cd0-9a7f-99f13d5b1606 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:59.615975 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1075/2144] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:38:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:38:59.756065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:38:59.764341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273290e2-5cc2-46e9-8393-2868f2040b98 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:59.764633 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273290e2-5cc2-46e9-8393-2868f2040b98 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:59.764859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65143f4-18a0-47d6-b0ee-c45561a15ab1 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:38:59.764953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273290e2-5cc2-46e9-8393-2868f2040b98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:59.768551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:38:59.771312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:38:59.771312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:38:59.771312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:38:59.780762 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:38:59.782231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:38:59.793974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1070/2145] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:38:59 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:38:59.796979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-273290e2-5cc2-46e9-8393-2868f2040b98 None None] MFA Rules not processed for user `14729da7649a4ab785ff87981968d342`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:38:59.798238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-273290e2-5cc2-46e9-8393-2868f2040b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:38:59.807662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1076/2146] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:38:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:00.083672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:00.085080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:00.147608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecec113f-5daf-4d39-b532-d3c720664847 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:00.152876 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:00.153983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:00.154278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:00.154564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:00.156547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:00.167133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:00.168576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:00.177726 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1071/2147] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:00.395141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273290e2-5cc2-46e9-8393-2868f2040b98 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:00.414859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273290e2-5cc2-46e9-8393-2868f2040b98 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:00.419885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:00.420537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:00.420794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:00.421067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:00.509763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:00.511545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:00.520817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1077/2148] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:00 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:04.627562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:04.638154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c6ce9b-a23b-401f-a3ec-fee5e0e81eb3 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:04.642083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72abb95a-a395-4478-81ee-f88217a8bf85 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:04.642712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72abb95a-a395-4478-81ee-f88217a8bf85 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:04.642983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72abb95a-a395-4478-81ee-f88217a8bf85 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:04.643328 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72abb95a-a395-4478-81ee-f88217a8bf85 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:04.646162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-72abb95a-a395-4478-81ee-f88217a8bf85 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:04.714634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72abb95a-a395-4478-81ee-f88217a8bf85 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:04.716074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72abb95a-a395-4478-81ee-f88217a8bf85 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:04.726268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1072/2149] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:04 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:05.756692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:05.784262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:05.829269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd02d7a3-7867-4415-90b4-a8d546476aa7 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:05.834483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:05.835144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:05.835498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:05.835847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] PATH_INFO: `/v3/users/8c2e59073d7b44db8d32417e742c489f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:05.837068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c2e59073d7b44db8d32417e742c489f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:05.839158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:06.048468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c2e59073d7b44db8d32417e742c489f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:06.049149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c2e59073d7b44db8d32417e742c489f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:06.061610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c2e59073d7b44db8d32417e742c489f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:06.069583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c2e59073d7b44db8d32417e742c489f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:06.070242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c2e59073d7b44db8d32417e742c489f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:06.163734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1078/2150] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:05 2022] DELETE /identity/v3/users/8c2e59073d7b44db8d32417e742c489f => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:06.169335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:06.169813 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:06.170108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:06.191225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:06.193045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:06.269656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1073/2151] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:06.279713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:06.281174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:06.342173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174d83d3-1b02-4d88-a3af-47f4cdfd5e2a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:06.345892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:06.346505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:06.346811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:06.347435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:06.356944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:06.358509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:06.369610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1079/2152] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:06.816741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:06.816741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:06.838461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763c3fce-ccc8-411e-92c0-144af9ed71e9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:06.845742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:06.845742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:06.845742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:06.845742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] PATH_INFO: `/v3/projects/73502baaf3ba4f26894cc22e3978dc59` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:06.846271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] RBAC: Authorizing `identity:delete_project(project_id=73502baaf3ba4f26894cc22e3978dc59)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:06.848465 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:06.859939 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:06.974291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73502baaf3ba4f26894cc22e3978dc59', 'request_id': 'req-61ccc3ce-2825-4901-9155-6bffc1a6137b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.041234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73502baaf3ba4f26894cc22e3978dc59', 'request_id': 'req-61ccc3ce-2825-4901-9155-6bffc1a6137b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.048308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73502baaf3ba4f26894cc22e3978dc59', 'request_id': 'req-61ccc3ce-2825-4901-9155-6bffc1a6137b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.059232 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] The token cache is being invalidate because project 73502baaf3ba4f26894cc22e3978dc59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:07.059529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.063050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.063050 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1074/2153] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:06 2022] DELETE /identity/v3/projects/73502baaf3ba4f26894cc22e3978dc59 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:07.065884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:07.075488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d4b159-3dca-482d-8258-f3046c731f47 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:07.122541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:07.137694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:07.138323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:07.138606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:07.138881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] PATH_INFO: `/v3/users/7a51768350e14f51947042a65af1df74` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:07.139996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a51768350e14f51947042a65af1df74)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:07.141983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:07.323032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a51768350e14f51947042a65af1df74'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.323621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a51768350e14f51947042a65af1df74'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.335875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a51768350e14f51947042a65af1df74'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.342738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a51768350e14f51947042a65af1df74'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.343383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-56e533bc-4eb9-4a30-8898-23201c874523 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a51768350e14f51947042a65af1df74'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:07.400566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1080/2154] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:07 2022] DELETE /identity/v3/users/7a51768350e14f51947042a65af1df74 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:07.884475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:07.886063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:07.909459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccc3ce-2825-4901-9155-6bffc1a6137b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:07.991743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:07.992461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:07.992760 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:07.993026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] PATH_INFO: `/v3/projects/3b0f76eba64d4597b254c0cfb1f504f0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:07.994191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b0f76eba64d4597b254c0cfb1f504f0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:07.996091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:08.145596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b0f76eba64d4597b254c0cfb1f504f0', 'request_id': 'req-b96b2a92-6822-46f9-88af-54b65c6a513b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:08.189932 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b0f76eba64d4597b254c0cfb1f504f0', 'request_id': 'req-b96b2a92-6822-46f9-88af-54b65c6a513b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:08.197429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b0f76eba64d4597b254c0cfb1f504f0', 'request_id': 'req-b96b2a92-6822-46f9-88af-54b65c6a513b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:08.209398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] The token cache is being invalidate because project 3b0f76eba64d4597b254c0cfb1f504f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:08.209398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:08.210647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b96b2a92-6822-46f9-88af-54b65c6a513b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:08.211720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1075/2155] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:07 2022] DELETE /identity/v3/projects/3b0f76eba64d4597b254c0cfb1f504f0 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:08.216696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015b72c-b5cb-4ebe-a756-239066cfe000 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.217180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015b72c-b5cb-4ebe-a756-239066cfe000 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.217298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015b72c-b5cb-4ebe-a756-239066cfe000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.245382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c015b72c-b5cb-4ebe-a756-239066cfe000 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:08.246639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c015b72c-b5cb-4ebe-a756-239066cfe000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:08.313763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1081/2156] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:08.321878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a8f5e9-4a36-468c-9e0c-83cf012ef992 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.322163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a8f5e9-4a36-468c-9e0c-83cf012ef992 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.322541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a8f5e9-4a36-468c-9e0c-83cf012ef992 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.349239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-60a8f5e9-4a36-468c-9e0c-83cf012ef992 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:08.350011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-60a8f5e9-4a36-468c-9e0c-83cf012ef992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:08.358349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1076/2157] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:08.364403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015b72c-b5cb-4ebe-a756-239066cfe000 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:08.376390 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015b72c-b5cb-4ebe-a756-239066cfe000 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:08.382545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:08.383170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.383450 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.383724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.384379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:39:08.384680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:08.386497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:08.401180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1082/2158] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:39:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:39:08.406423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a8f5e9-4a36-468c-9e0c-83cf012ef992 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:08.418234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a8f5e9-4a36-468c-9e0c-83cf012ef992 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:08.424589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:08.425986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.426279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.426558 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.427955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:08.430394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:08.528725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:08.537834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1077/2159] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:08 2022] POST /identity/v3/projects => generated 385 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:08.542761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:08.555107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d55689-cfca-4e78-b6ea-df1c90413ffe admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:08.633875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:08.655153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:08.656635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.657679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.658078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.658838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:08.661679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:08.781397 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1083/2160] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:08 2022] POST /identity/v3/users => generated 293 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:08.786659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:08.798227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5075e815-cf16-47e1-b56f-49bf8553b24b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:08.803977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:08.804610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.804905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.805181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.805958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:08.807500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:08.824704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1078/2161] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:08 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:08.825974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:08.839365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a714121-bbb5-4b7d-a921-4f7d94af1544 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:08.845036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1255d9be-d54c-41a6-9231-ce53a97608a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:08.848742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1255d9be-d54c-41a6-9231-ce53a97608a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.852251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1255d9be-d54c-41a6-9231-ce53a97608a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.852933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1255d9be-d54c-41a6-9231-ce53a97608a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.853678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1255d9be-d54c-41a6-9231-ce53a97608a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:08.855396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1255d9be-d54c-41a6-9231-ce53a97608a3 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:08.869396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1084/2162] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:08 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:08.874071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:08.886081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64e425d-c05d-46a3-b35e-b652c91c6ec7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:08.891468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:08.892743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.893108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.893417 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] PATH_INFO: `/v3/projects/12a0d5ad315745b790ce0cdc6cceb1da/users/58facc5b076d469c9166bdd457d05bce/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:08.911487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=12a0d5ad315745b790ce0cdc6cceb1da, user_id=58facc5b076d469c9166bdd457d05bce, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:08.914344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:08.998809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1079/2163] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:08 2022] PUT /identity/v3/projects/12a0d5ad315745b790ce0cdc6cceb1da/users/58facc5b076d469c9166bdd457d05bce/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:08.999227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4ada2-abe9-4091-b9f1-035016e46015 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:08.999373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4ada2-abe9-4091-b9f1-035016e46015 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:08.999709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4ada2-abe9-4091-b9f1-035016e46015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.040168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a4f4ada2-abe9-4091-b9f1-035016e46015 None None] MFA Rules not processed for user `58facc5b076d469c9166bdd457d05bce`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:09.041718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a4f4ada2-abe9-4091-b9f1-035016e46015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:09.120669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1085/2164] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:09.125461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:09.208837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b506d36-21e0-4abd-8c80-2a83cb7b95ed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:09.220311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:09.220955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:09.221234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:09.221794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.222845 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:09.226268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:09.338493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1080/2165] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:09 2022] POST /identity/v3/projects => generated 387 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:09.339629 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4ada2-abe9-4091-b9f1-035016e46015 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:09.350626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4ada2-abe9-4091-b9f1-035016e46015 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:09.414557 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:09.415174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:09.415456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:09.415726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.416386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:09.418373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:09.544619 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1086/2166] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:09 2022] POST /identity/v3/users => generated 294 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:09.548581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:09.559656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd589ab8-ecf9-441f-abf5-ca986032e988 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:09.564239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:09.564840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:09.565129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:09.565439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.566229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:09.567835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:09.578502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1081/2167] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:09.582880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:09.593056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c85b5-6748-43ac-b09e-745ccad7d0ff admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:09.598463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e216408b-2da5-4d22-a5de-658f182397fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:09.599060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216408b-2da5-4d22-a5de-658f182397fd admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:09.599343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216408b-2da5-4d22-a5de-658f182397fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:09.599626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216408b-2da5-4d22-a5de-658f182397fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.600288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e216408b-2da5-4d22-a5de-658f182397fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:09.601934 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e216408b-2da5-4d22-a5de-658f182397fd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:09.613184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1087/2168] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:09.617698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:09.627701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f051daa-9c11-4e66-aaff-f714b7fc4cfd admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:09.632393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:09.638427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:09.638427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:09.638427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] PATH_INFO: `/v3/projects/3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1/users/e9437c77b2fc4cc19cd5eeeb1db11623/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.649341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1, user_id=e9437c77b2fc4cc19cd5eeeb1db11623, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:09.651988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:09.744165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1082/2169] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:09 2022] PUT /identity/v3/projects/3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1/users/e9437c77b2fc4cc19cd5eeeb1db11623/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:09.755839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab247339-160d-49d6-9b99-abaa4b167357 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:09.756125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab247339-160d-49d6-9b99-abaa4b167357 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:09.756399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab247339-160d-49d6-9b99-abaa4b167357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.785561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ab247339-160d-49d6-9b99-abaa4b167357 None None] MFA Rules not processed for user `e9437c77b2fc4cc19cd5eeeb1db11623`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:09.787007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ab247339-160d-49d6-9b99-abaa4b167357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:09.814512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:09.852690 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1088/2170] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:09 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:09.856747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:09.939335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afd97dc-cc28-4279-ad3e-72782e604fba admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:09.944601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:09.945239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:09.945771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:09.945894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:09.946553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:09.948468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:10.037823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:10.044422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1083/2171] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:09 2022] POST /identity/v3/projects => generated 387 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:10.048762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab247339-160d-49d6-9b99-abaa4b167357 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:10.060231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab247339-160d-49d6-9b99-abaa4b167357 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:10.120798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:10.121438 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:10.121724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:10.121998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:10.122624 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:10.124548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:10.213709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1089/2172] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:10 2022] POST /identity/v3/users => generated 300 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:10.217921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:10.228899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b22961-2fdc-426c-b691-65b5eece9a47 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:10.233580 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:10.234195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:10.234526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:10.234803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:10.235451 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:10.237863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:10.257732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1084/2173] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:10.260742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:10.271152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e569c74-4097-4979-8b29-c652256d618b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:10.275927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:10.286365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:10.286365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:10.286365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] PATH_INFO: `/v3/projects/e033baadf846417386231f7c9da334a6/users/7927c650ba794afbad468d7ff58be823/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:10.293590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e033baadf846417386231f7c9da334a6, user_id=7927c650ba794afbad468d7ff58be823, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:10.296253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:10.379197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1090/2174] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:10 2022] PUT /identity/v3/projects/e033baadf846417386231f7c9da334a6/users/7927c650ba794afbad468d7ff58be823/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:10.383419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:10.393454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1dc73-5724-47e2-8c0b-ccd94041f360 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:10.451180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8291724-9eaf-43a5-9194-7ba3b0afb4ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:10.451796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8291724-9eaf-43a5-9194-7ba3b0afb4ec admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:10.452084 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8291724-9eaf-43a5-9194-7ba3b0afb4ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:10.452365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8291724-9eaf-43a5-9194-7ba3b0afb4ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:10.453006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8291724-9eaf-43a5-9194-7ba3b0afb4ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:10.454660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8291724-9eaf-43a5-9194-7ba3b0afb4ec admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:10.465148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1085/2175] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:10.469254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:10.479738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d977fb-a1c3-487f-b28d-bdc744cf1ca7 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:10.484316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:10.484960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:10.485242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:10.485553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] PATH_INFO: `/v3/projects/e033baadf846417386231f7c9da334a6/users/7927c650ba794afbad468d7ff58be823/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:10.488047 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] RBAC: Authorizing `identity:create_grant(project_id=e033baadf846417386231f7c9da334a6, user_id=7927c650ba794afbad468d7ff58be823, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:10.490735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:10.501215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1091/2176] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:10 2022] PUT /identity/v3/projects/e033baadf846417386231f7c9da334a6/users/7927c650ba794afbad468d7ff58be823/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:10.511979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f73b8b8-07ad-4db9-805a-9c59707a4c4c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:10.512348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f73b8b8-07ad-4db9-805a-9c59707a4c4c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:10.512631 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f73b8b8-07ad-4db9-805a-9c59707a4c4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:10.540796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0f73b8b8-07ad-4db9-805a-9c59707a4c4c None None] MFA Rules not processed for user `7927c650ba794afbad468d7ff58be823`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:10.542038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0f73b8b8-07ad-4db9-805a-9c59707a4c4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:10.612091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1086/2177] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:10 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:10.623331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:10.625029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:10.684668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e2df3-863b-4d09-93ff-41d542ca0bce admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:10.688551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:10.689154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:10.689469 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:10.689740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:10.699525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:10.700950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:10.710310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1092/2178] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:10 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:11.858893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f73b8b8-07ad-4db9-805a-9c59707a4c4c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:11.869916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f73b8b8-07ad-4db9-805a-9c59707a4c4c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:11.873502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:11.874139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:11.874420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:11.874757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:11.933241 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:11.935604 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:11.945786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1087/2179] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:11 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:20.706049 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=85,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:21.941831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:22.393716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:22.402186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e938f1-fc45-4a2c-87c2-8465db6d3364 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:22.405844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba33775-5eaf-48f3-a9b7-47c7ad8a3825 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:22.406434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba33775-5eaf-48f3-a9b7-47c7ad8a3825 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:22.406710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba33775-5eaf-48f3-a9b7-47c7ad8a3825 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:22.406979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba33775-5eaf-48f3-a9b7-47c7ad8a3825 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:22.462896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba33775-5eaf-48f3-a9b7-47c7ad8a3825 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:22.464258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba33775-5eaf-48f3-a9b7-47c7ad8a3825 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:22.474586 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1093/2180] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:22 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:29.905978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:29.971567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1f709a-183b-4a45-b571-93c762888d3c service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:29.976635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:29.977340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:29.977641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:29.977944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] PATH_INFO: `/v3/users/58facc5b076d469c9166bdd457d05bce` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:29.979069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58facc5b076d469c9166bdd457d05bce)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:29.981103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:30.170197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58facc5b076d469c9166bdd457d05bce'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:30.191779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58facc5b076d469c9166bdd457d05bce'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:30.264143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58facc5b076d469c9166bdd457d05bce'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:30.264776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58facc5b076d469c9166bdd457d05bce'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:30.265428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58facc5b076d469c9166bdd457d05bce'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:30.282782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1088/2181] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:29 2022] DELETE /identity/v3/users/58facc5b076d469c9166bdd457d05bce => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:30.291784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:30.291784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:30.295153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:30.315890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:30.317087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:30.403847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1094/2182] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:30.405022 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:30.406505 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:30.465554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b57433-bb17-4120-b94f-0d3c37199525 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:30.469230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:30.469874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:30.470160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:30.470441 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:30.480431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:30.481904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:30.492148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1089/2183] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:30.922809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:30.924354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:30.937930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa0e9b8-2819-4139-bf2c-c426a26cc78f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:30.942980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:30.943598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:30.943949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:30.944439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] PATH_INFO: `/v3/projects/12a0d5ad315745b790ce0cdc6cceb1da` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:30.945759 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] RBAC: Authorizing `identity:delete_project(project_id=12a0d5ad315745b790ce0cdc6cceb1da)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:30.948270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:31.080969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12a0d5ad315745b790ce0cdc6cceb1da', 'request_id': 'req-1f9670b7-2d73-4af9-851d-bc230dc9152b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.087432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12a0d5ad315745b790ce0cdc6cceb1da', 'request_id': 'req-1f9670b7-2d73-4af9-851d-bc230dc9152b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.094885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12a0d5ad315745b790ce0cdc6cceb1da', 'request_id': 'req-1f9670b7-2d73-4af9-851d-bc230dc9152b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.166701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] The token cache is being invalidate because project 12a0d5ad315745b790ce0cdc6cceb1da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:31.166985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.167599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.169076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1095/2184] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:30 2022] DELETE /identity/v3/projects/12a0d5ad315745b790ce0cdc6cceb1da => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:31.173135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:31.190880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cbd714-ba4f-4780-ad04-699a7afa9305 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:31.202597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:31.241385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:31.242018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:31.242302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:31.242578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] PATH_INFO: `/v3/users/e9437c77b2fc4cc19cd5eeeb1db11623` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:31.243699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9437c77b2fc4cc19cd5eeeb1db11623)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:31.245641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:31.473619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9437c77b2fc4cc19cd5eeeb1db11623'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.485910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9437c77b2fc4cc19cd5eeeb1db11623'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.572846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9437c77b2fc4cc19cd5eeeb1db11623'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.573513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9437c77b2fc4cc19cd5eeeb1db11623'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.574134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9437c77b2fc4cc19cd5eeeb1db11623'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:31.588488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1090/2185] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:31 2022] DELETE /identity/v3/users/e9437c77b2fc4cc19cd5eeeb1db11623 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:32.015543 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:32.017175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:32.038606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9670b7-2d73-4af9-851d-bc230dc9152b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:32.058329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:32.096420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:32.097093 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:32.097407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:32.097692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] PATH_INFO: `/v3/projects/3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:32.098790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:32.100924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:32.218635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1', 'request_id': 'req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.224082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1', 'request_id': 'req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.230751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1', 'request_id': 'req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.452555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] The token cache is being invalidate because project 3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:32.452872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.453517 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.455010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1096/2186] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:32 2022] DELETE /identity/v3/projects/3e8e6cbfed0e4fb08e1dbe7f0b9fc8c1 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:32.459029 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:32.460435 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:32.473191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec32da87-3f00-40fa-95a2-a42b86b08ba8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:32.533976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:32.534618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:32.534906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:32.535182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] PATH_INFO: `/v3/users/7927c650ba794afbad468d7ff58be823` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:32.536352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7927c650ba794afbad468d7ff58be823)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:32.538313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:32.697405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:32.703381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:32.704976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:32.707355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7927c650ba794afbad468d7ff58be823'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.711329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceec0ee6-54c5-499b-bf1d-d8c6d86507eb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:32.715983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:32.716607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:32.716890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:32.717161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] PATH_INFO: `/v3/users/ce72c24b489a4d929545e400d73f94ce` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:32.718295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce72c24b489a4d929545e400d73f94ce)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:32.720243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7927c650ba794afbad468d7ff58be823'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.720371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:32.790439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7927c650ba794afbad468d7ff58be823'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.791091 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7927c650ba794afbad468d7ff58be823'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.791740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a238e6df-0c3d-4af3-b71f-62dfc355df03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7927c650ba794afbad468d7ff58be823'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.806258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1091/2187] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:32 2022] DELETE /identity/v3/users/7927c650ba794afbad468d7ff58be823 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:32.873432 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce72c24b489a4d929545e400d73f94ce'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.874110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce72c24b489a4d929545e400d73f94ce'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.887567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce72c24b489a4d929545e400d73f94ce'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.894560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce72c24b489a4d929545e400d73f94ce'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:32.899219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce72c24b489a4d929545e400d73f94ce'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.009788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1097/2188] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:32 2022] DELETE /identity/v3/users/ce72c24b489a4d929545e400d73f94ce => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:33.014279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.014574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.015066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.036666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:33.038321 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:33.114528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1092/2189] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:33.124779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:33.126244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:33.155663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:33.185310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4928be-55c3-4f5c-be56-8b721787cbc8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:33.188909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c2e35-9da1-4a07-918e-5b0940197ab2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:33.189553 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c2e35-9da1-4a07-918e-5b0940197ab2 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.189854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c2e35-9da1-4a07-918e-5b0940197ab2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.190108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c2e35-9da1-4a07-918e-5b0940197ab2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.199662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54c2e35-9da1-4a07-918e-5b0940197ab2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:33.201119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54c2e35-9da1-4a07-918e-5b0940197ab2 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:33.210840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1098/2190] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:39:33.260912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:33.263907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:33.282409 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783dce45-1c5c-45ac-8b62-dd6e3510b646 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:33.288111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:33.289615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.290001 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.290181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] PATH_INFO: `/v3/projects/e033baadf846417386231f7c9da334a6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.291464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e033baadf846417386231f7c9da334a6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:33.293568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:33.414148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e033baadf846417386231f7c9da334a6', 'request_id': 'req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.532839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e033baadf846417386231f7c9da334a6', 'request_id': 'req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.543105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e033baadf846417386231f7c9da334a6', 'request_id': 'req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.555574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] The token cache is being invalidate because project e033baadf846417386231f7c9da334a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:33.555867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.556550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.558305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1093/2191] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:33 2022] DELETE /identity/v3/projects/e033baadf846417386231f7c9da334a6 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:33.566083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9833125-c84d-4b8b-9696-5ab9a1cc045b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.566083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9833125-c84d-4b8b-9696-5ab9a1cc045b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.566083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9833125-c84d-4b8b-9696-5ab9a1cc045b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.585120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d9833125-c84d-4b8b-9696-5ab9a1cc045b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:33.589393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d9833125-c84d-4b8b-9696-5ab9a1cc045b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:33.611475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:33.613171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:33.657247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1099/2192] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:33.667851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494495ff-7e70-4ca3-879a-7965256b1d7b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.668137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494495ff-7e70-4ca3-879a-7965256b1d7b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.668414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494495ff-7e70-4ca3-879a-7965256b1d7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.674264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:33.689082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-494495ff-7e70-4ca3-879a-7965256b1d7b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:33.690315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-494495ff-7e70-4ca3-879a-7965256b1d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:33.693987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db6773-4dfd-4d5a-ba48-4ea992294b4e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:33.698447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1100/2193] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:33.699891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:33.700568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.700852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.701131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] PATH_INFO: `/v3/projects/e77ed36e4a894aa89c375a13af93a71b` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.702306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e77ed36e4a894aa89c375a13af93a71b)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:33.702604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494495ff-7e70-4ca3-879a-7965256b1d7b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:33.704226 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:33.711661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494495ff-7e70-4ca3-879a-7965256b1d7b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:33.716337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:33.716931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.717213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.717529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.718173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:39:33.718506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:33.720120 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:33.732675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1101/2194] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:39:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:39:33.737052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:33.748213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfefa68-4732-42e4-93ca-af1e792e01d4 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:33.754077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:33.754602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.754882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.755155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.755776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:33.757833 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:33.817512 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e77ed36e4a894aa89c375a13af93a71b', 'request_id': 'req-33dc5cc2-7543-482c-9116-79cc51c37ddb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.887468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1102/2195] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:33 2022] POST /identity/v3/projects => generated 379 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:33.891654 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:33.904368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e77ed36e4a894aa89c375a13af93a71b', 'request_id': 'req-33dc5cc2-7543-482c-9116-79cc51c37ddb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.911620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e77ed36e4a894aa89c375a13af93a71b', 'request_id': 'req-33dc5cc2-7543-482c-9116-79cc51c37ddb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.923160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] The token cache is being invalidate because project e77ed36e4a894aa89c375a13af93a71b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:33.923568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.924413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:33.926077 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1094/2196] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:33 2022] DELETE /identity/v3/projects/e77ed36e4a894aa89c375a13af93a71b => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:33.930313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:33.931938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:33.956257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d9573a-b0a3-4824-a86a-7122a95bec6e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:33.961754 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:33.962524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:33.962816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:33.963099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:33.963763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:33.965867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.002449 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dc5cc2-7543-482c-9116-79cc51c37ddb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.014224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.014224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.014224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.014224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] PATH_INFO: `/v3/users/2b8a548ef8e044758fe2d28702bf08bb` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.014573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b8a548ef8e044758fe2d28702bf08bb)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.014573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.050027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1103/2197] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:33 2022] POST /identity/v3/users => generated 290 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:34.057628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.060844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:34.074331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504193f5-2c9a-4704-a962-d46c3d21d7a2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.079413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.080147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.080444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.080724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.081409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.083072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.098062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1104/2198] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:34 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:34.099193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.110532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddf4ce-75ff-487b-be9d-82c04ba887ec admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.113213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:34.117261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.118017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.118370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.118679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.119322 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.120955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.132762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1105/2199] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:34.136954 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.146742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc28159-1850-4fae-8565-3f73fbf5ac1d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.152110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc544a79-29ab-440e-9b4a-965a00ceca56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.152730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc544a79-29ab-440e-9b4a-965a00ceca56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.153012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc544a79-29ab-440e-9b4a-965a00ceca56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.153285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc544a79-29ab-440e-9b4a-965a00ceca56 admin admin] PATH_INFO: `/v3/projects/2b2191df859c4ee29be70083c7ac50c6/users/632cc545e5284d0da817b36e7524ff7b/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.168278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc544a79-29ab-440e-9b4a-965a00ceca56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b2191df859c4ee29be70083c7ac50c6, user_id=632cc545e5284d0da817b36e7524ff7b, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.170996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc544a79-29ab-440e-9b4a-965a00ceca56 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.187251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b8a548ef8e044758fe2d28702bf08bb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:34.200171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b8a548ef8e044758fe2d28702bf08bb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:34.276591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1106/2200] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:34 2022] PUT /identity/v3/projects/2b2191df859c4ee29be70083c7ac50c6/users/632cc545e5284d0da817b36e7524ff7b/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:34.288080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f68c3dd-7d0a-43b1-83ce-412da9b68288 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.288394 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f68c3dd-7d0a-43b1-83ce-412da9b68288 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.289250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f68c3dd-7d0a-43b1-83ce-412da9b68288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.303410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b8a548ef8e044758fe2d28702bf08bb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:34.304074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b8a548ef8e044758fe2d28702bf08bb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:34.306037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b8a548ef8e044758fe2d28702bf08bb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:34.321906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5f68c3dd-7d0a-43b1-83ce-412da9b68288 None None] MFA Rules not processed for user `632cc545e5284d0da817b36e7524ff7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:34.322688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5f68c3dd-7d0a-43b1-83ce-412da9b68288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:34.324685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1095/2201] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:33 2022] DELETE /identity/v3/users/2b8a548ef8e044758fe2d28702bf08bb => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:34.400040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1107/2202] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:34 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:34.405120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.411648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:34.472937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deacd402-e93d-43ff-a2c8-f1b3b995d9c9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.477698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.478311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.478597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.478874 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.479503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.481430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.574991 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1096/2203] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:34 2022] POST /identity/v3/projects => generated 377 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:34.579560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f68c3dd-7d0a-43b1-83ce-412da9b68288 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.593971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f68c3dd-7d0a-43b1-83ce-412da9b68288 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.659096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.659713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.659992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.660266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.660904 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.662920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.837485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:34.843481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1108/2204] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:34 2022] POST /identity/v3/users => generated 289 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:34.845767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.847134 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.847760 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:34.858419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254a15f-a127-4fc3-9257-67b9cdc3834b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.860655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e11a99-b9a4-43ce-9716-b43ffe268f44 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.864805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.864967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.865192 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.865539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.866186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.867891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.868299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.868805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.869312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.874597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] PATH_INFO: `/v3/projects/21165136cdae4683b5f427a57e53ea97` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.874597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21165136cdae4683b5f427a57e53ea97)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.874597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.880539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1109/2205] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:34.885016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.895254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7fcb4-43f6-4eb7-91ad-25881039b84f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.900827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.901562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.903739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.903739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.903739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.904497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:34.915741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1110/2206] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:34.919791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:34.929651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c25510-d1d2-4b3c-94fb-014751825049 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:34.934445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85da668-9a82-49d7-bb0f-c902da274981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:34.935091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85da668-9a82-49d7-bb0f-c902da274981 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:34.935358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85da668-9a82-49d7-bb0f-c902da274981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:34.935645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85da668-9a82-49d7-bb0f-c902da274981 admin admin] PATH_INFO: `/v3/projects/9cb5688269aa41548705c8e7d6f8396f/users/61858605a1d74b9c8efc4fe7c54a20dd/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:34.951900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85da668-9a82-49d7-bb0f-c902da274981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cb5688269aa41548705c8e7d6f8396f, user_id=61858605a1d74b9c8efc4fe7c54a20dd, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:34.954703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85da668-9a82-49d7-bb0f-c902da274981 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.019563 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1111/2207] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:34 2022] PUT /identity/v3/projects/9cb5688269aa41548705c8e7d6f8396f/users/61858605a1d74b9c8efc4fe7c54a20dd/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:35.027964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21165136cdae4683b5f427a57e53ea97', 'request_id': 'req-1d98bdbf-933c-4559-b292-55c5e727f702'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:35.031425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.031787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.032152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.039751 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:35.060360 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] MFA Rules not processed for user `61858605a1d74b9c8efc4fe7c54a20dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:35.061602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:35.075486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21165136cdae4683b5f427a57e53ea97', 'request_id': 'req-1d98bdbf-933c-4559-b292-55c5e727f702'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:35.084063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21165136cdae4683b5f427a57e53ea97', 'request_id': 'req-1d98bdbf-933c-4559-b292-55c5e727f702'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:35.098261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] The token cache is being invalidate because project 21165136cdae4683b5f427a57e53ea97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:35.098622 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:35.099379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1d98bdbf-933c-4559-b292-55c5e727f702 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:35.101367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1097/2208] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:34 2022] DELETE /identity/v3/projects/21165136cdae4683b5f427a57e53ea97 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:35.106127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b94ef5-6fdc-4f16-9f43-b7c8c318416b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.106445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b94ef5-6fdc-4f16-9f43-b7c8c318416b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.106727 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b94ef5-6fdc-4f16-9f43-b7c8c318416b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.131891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-14b94ef5-6fdc-4f16-9f43-b7c8c318416b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:35.131891 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-14b94ef5-6fdc-4f16-9f43-b7c8c318416b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:35.145468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1112/2209] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:35.148621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.151644 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:35.218218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1098/2210] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:35.223947 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e791216-fef7-4bee-a367-2e767019a1ba None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.226053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3767ca-ccd3-45dc-b522-47c4651d9b82 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.226362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3767ca-ccd3-45dc-b522-47c4651d9b82 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.226644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3767ca-ccd3-45dc-b522-47c4651d9b82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.229267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.229906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.230240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.230519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.231171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.233108 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.234226 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:35.246626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5e3767ca-ccd3-45dc-b522-47c4651d9b82 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:35.247797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5e3767ca-ccd3-45dc-b522-47c4651d9b82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:35.255779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1099/2211] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:35.260461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3767ca-ccd3-45dc-b522-47c4651d9b82 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.269381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3767ca-ccd3-45dc-b522-47c4651d9b82 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.274576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.275156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.275434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.275706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.276340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:39:35.276640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.278247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.290731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1100/2212] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:39:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:39:35.294848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.305097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e034c87-ea46-4cd6-8040-75f5bc9fc328 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.309978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.310564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.310848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.311129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.311748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.313677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.323395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1113/2213] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/projects => generated 377 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:35.333638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.399018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1101/2214] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/projects => generated 365 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:35.403315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.406090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6ee828-a280-402f-96bc-f058d7d2dddd admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.427448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:35.466906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.467545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.467829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.468111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.468881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.470860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.472348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dfd7d5-9cca-406b-9680-c82ae80896b8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.478141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.478703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.487189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.487189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.487189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.487189 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.578241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1114/2215] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/users => generated 295 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:35.582030 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.592041 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60056bab-ceae-4226-95d9-83a2a0b4f09a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.596708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.597296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.597613 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.597892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.598656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.600295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.608051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1102/2216] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/users => generated 283 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:35.611201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1115/2217] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:35.612000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.615269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.623236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b076b0-e41a-4c65-b371-2a97e109f148 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.626102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e20854-6324-4000-b9cb-e4ce1b14227e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.628097 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.628702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.628990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.629383 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.629954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.630888 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.631533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.631664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.631883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.632245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] PATH_INFO: `/v3/projects/a9883b2b496a47f4a66d4adca797c108/users/e0035f9c629148d0a7b48b29bfe8f049/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.643083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1103/2218] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:35.646963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.649155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9883b2b496a47f4a66d4adca797c108, user_id=e0035f9c629148d0a7b48b29bfe8f049, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.651943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.658809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495182ca-f3ea-415a-9668-180c55114998 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.664083 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.664814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.665119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.665574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.666214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.667842 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.681283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1104/2219] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:35.685540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.696599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4073b-00f3-4976-8b8a-497b1b10cd7f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.701366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93783bed-1b36-4189-9d6a-4dc04ae2d1f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.702032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93783bed-1b36-4189-9d6a-4dc04ae2d1f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.702317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93783bed-1b36-4189-9d6a-4dc04ae2d1f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.702596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93783bed-1b36-4189-9d6a-4dc04ae2d1f3 admin admin] PATH_INFO: `/v3/projects/e944162cff6b4331bb86199026e95a0a/users/5492b14d194a4e86b14398d559d6e1e4/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.716706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1116/2220] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:35 2022] PUT /identity/v3/projects/a9883b2b496a47f4a66d4adca797c108/users/e0035f9c629148d0a7b48b29bfe8f049/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:35.720324 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93783bed-1b36-4189-9d6a-4dc04ae2d1f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e944162cff6b4331bb86199026e95a0a, user_id=5492b14d194a4e86b14398d559d6e1e4, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.720907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93783bed-1b36-4189-9d6a-4dc04ae2d1f3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.722958 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.778638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1105/2221] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:35 2022] PUT /identity/v3/projects/e944162cff6b4331bb86199026e95a0a/users/5492b14d194a4e86b14398d559d6e1e4/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:35.786472 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d964a-5f14-4eea-9885-d07331087f7f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.793000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a212478-1cb2-4cae-8d9a-b0ad108b7272 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.793292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a212478-1cb2-4cae-8d9a-b0ad108b7272 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.793639 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a212478-1cb2-4cae-8d9a-b0ad108b7272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.822532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5a212478-1cb2-4cae-8d9a-b0ad108b7272 None None] MFA Rules not processed for user `5492b14d194a4e86b14398d559d6e1e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:35.823745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5a212478-1cb2-4cae-8d9a-b0ad108b7272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:35.845568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.846309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.846699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.847080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.847827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.849591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.860513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1117/2222] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:35.864875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.875881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4664c-bd9d-45e1-9b2e-b96fcc54300a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.880447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2733b9-1bad-4efd-9773-af5726cc85b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.880921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:35.881121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2733b9-1bad-4efd-9773-af5726cc85b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.881396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2733b9-1bad-4efd-9773-af5726cc85b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.882714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2733b9-1bad-4efd-9773-af5726cc85b7 admin admin] PATH_INFO: `/v3/projects/a9883b2b496a47f4a66d4adca797c108/users/e0035f9c629148d0a7b48b29bfe8f049/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.884295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2733b9-1bad-4efd-9773-af5726cc85b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9883b2b496a47f4a66d4adca797c108, user_id=e0035f9c629148d0a7b48b29bfe8f049, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.886950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2733b9-1bad-4efd-9773-af5726cc85b7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:35.891208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1106/2223] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:35.895412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a212478-1cb2-4cae-8d9a-b0ad108b7272 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:35.897918 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1118/2224] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:35 2022] PUT /identity/v3/projects/a9883b2b496a47f4a66d4adca797c108/users/e0035f9c629148d0a7b48b29bfe8f049/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:35.906035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a212478-1cb2-4cae-8d9a-b0ad108b7272 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:35.909306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.909620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.909897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.930170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:35.938675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] MFA Rules not processed for user `e0035f9c629148d0a7b48b29bfe8f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:35.939892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:35.966724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:35.967342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:35.967625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:35.967902 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:35.968535 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:35.970455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.013001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1119/2225] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:36.025594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.026966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:36.057963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1107/2226] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:35 2022] POST /identity/v3/projects => generated 363 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:36.058942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.086471 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99805d5-c89b-433c-9319-dc8cdce70fd2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.124106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274f32d2-7d8d-4c60-a02c-10ff6a1da8f2 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.128917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.129555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.129841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.130116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.130779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.133601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.136890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.137541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.137829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.138107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.199854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.201358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.202896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1108/2227] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:36 2022] POST /identity/v3/users => generated 288 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:36.206672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.207993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:36.211964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1120/2228] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:36 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:39:36.219291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfbf63a-247d-410e-a8e5-d672082d5848 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.223974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.224543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.224823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.225100 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.225789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.227673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.239161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1109/2229] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:36.245601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.256237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff08914-908b-44a3-a72c-c520883bd6cb service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.261717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.262443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.262724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.263002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] PATH_INFO: `/v3/projects/441563a8d2944235b43b77009b434105/users/403ef609de9f469ba82940d1803bf1c6/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.278903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=441563a8d2944235b43b77009b434105, user_id=403ef609de9f469ba82940d1803bf1c6, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.281703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.296816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.298576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:36.311801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49541fb8-96dc-4565-afdd-0212aa9cdc3f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.315695 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.316246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.316526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.316805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.318634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:36.367335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1121/2230] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:36 2022] PUT /identity/v3/projects/441563a8d2944235b43b77009b434105/users/403ef609de9f469ba82940d1803bf1c6/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:36.371323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.376626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:36.379717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.381213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.391262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1110/2231] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:36 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:36.440384 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a829871f-c4b1-4695-b92a-a18ce43d0a9c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.445965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8d282e-1f20-4336-bc6e-2966ecd2906f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.446578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8d282e-1f20-4336-bc6e-2966ecd2906f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.446863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8d282e-1f20-4336-bc6e-2966ecd2906f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.447162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8d282e-1f20-4336-bc6e-2966ecd2906f admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.447788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8d282e-1f20-4336-bc6e-2966ecd2906f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.449434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8d282e-1f20-4336-bc6e-2966ecd2906f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.459486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1122/2232] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:36 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:39:36.464722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.475760 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6663f927-f55c-485e-b484-b0890b750208 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.480534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.483908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.483908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.483908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] PATH_INFO: `/v3/projects/441563a8d2944235b43b77009b434105/users/403ef609de9f469ba82940d1803bf1c6/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.483908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] RBAC: Authorizing `identity:create_grant(project_id=441563a8d2944235b43b77009b434105, user_id=403ef609de9f469ba82940d1803bf1c6, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.487130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.497702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1111/2233] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:36 2022] PUT /identity/v3/projects/441563a8d2944235b43b77009b434105/users/403ef609de9f469ba82940d1803bf1c6/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:36.511969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80499599-ad4a-4c24-98b1-0cc6bb3ccc0f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.512342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80499599-ad4a-4c24-98b1-0cc6bb3ccc0f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.512642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80499599-ad4a-4c24-98b1-0cc6bb3ccc0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.545935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-80499599-ad4a-4c24-98b1-0cc6bb3ccc0f None None] MFA Rules not processed for user `403ef609de9f469ba82940d1803bf1c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:36.547286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-80499599-ad4a-4c24-98b1-0cc6bb3ccc0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:36.619740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1123/2234] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:36 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:36.630936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:36.689378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5089b2-b890-439e-a0ca-eed290dcf89c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:36.693090 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:36.693732 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:36.694032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:36.694307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:36.703663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:36.705080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:36.714676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1112/2235] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:36 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:39:37.035309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80499599-ad4a-4c24-98b1-0cc6bb3ccc0f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:37.045698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80499599-ad4a-4c24-98b1-0cc6bb3ccc0f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:37.049197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:37.049831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:37.050110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:37.050383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:37.106693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:37.108943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:37.110416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:37.121028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1124/2236] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:37 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:39:44.680244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:44.714844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:44.751841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae34328-656d-49c8-955e-c6d3eb284b63 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:44.756470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:44.757114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:44.757648 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:44.758033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] PATH_INFO: `/v3/users/5492b14d194a4e86b14398d559d6e1e4` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:44.759236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5492b14d194a4e86b14398d559d6e1e4)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:44.761211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:44.780876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3770035-8006-477c-bdc2-65625d83f07d service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:44.785770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:44.786578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:44.786863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:44.787137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] PATH_INFO: `/v3/users/632cc545e5284d0da817b36e7524ff7b` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:44.788811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] RBAC: Authorizing `identity:delete_user(user_id=632cc545e5284d0da817b36e7524ff7b)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:44.790883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:44.986368 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5492b14d194a4e86b14398d559d6e1e4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:44.997283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '632cc545e5284d0da817b36e7524ff7b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:44.997993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '632cc545e5284d0da817b36e7524ff7b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:44.999355 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5492b14d194a4e86b14398d559d6e1e4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:45.015429 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '632cc545e5284d0da817b36e7524ff7b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:45.018488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '632cc545e5284d0da817b36e7524ff7b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:45.019183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-fcc56ca8-9481-42d5-b355-7eea48de1768 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '632cc545e5284d0da817b36e7524ff7b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:45.091702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5492b14d194a4e86b14398d559d6e1e4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:45.092312 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5492b14d194a4e86b14398d559d6e1e4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:45.092906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b511e044-df0a-431d-aacd-7d67a71b533a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5492b14d194a4e86b14398d559d6e1e4'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:45.106393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1113/2237] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:44 2022] DELETE /identity/v3/users/5492b14d194a4e86b14398d559d6e1e4 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:45.110692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:45.111066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:45.111347 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:45.120131 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:45.133017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:45.134254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:45.154681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1125/2238] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:44 2022] DELETE /identity/v3/users/632cc545e5284d0da817b36e7524ff7b => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:45.159224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec509d18-a372-4e10-b340-119c62239bda None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:45.159512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec509d18-a372-4e10-b340-119c62239bda None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:45.159817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec509d18-a372-4e10-b340-119c62239bda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:45.178770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ec509d18-a372-4e10-b340-119c62239bda None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:45.181123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ec509d18-a372-4e10-b340-119c62239bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:45.193628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1114/2239] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:45.204181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:45.205616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:45.243681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1126/2240] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:45.253352 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec509d18-a372-4e10-b340-119c62239bda None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:45.254753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ec509d18-a372-4e10-b340-119c62239bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:45.264723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c32d0da-a300-492b-9c3b-c064e993d0d3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:45.273606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:45.313480 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec509d18-a372-4e10-b340-119c62239bda None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:45.315096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:45.315909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:45.316273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:45.316957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:45.318371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:45.318979 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:45.319263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:45.319537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:45.326122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:45.327560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:45.329240 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:45.330910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:45.337821 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1115/2241] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:39:45.343040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1127/2242] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:39:45.842740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:45.844227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:45.857499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e4af11-f4cc-45cd-94f4-84b907a38da4 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:45.862032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:45.863048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:45.863431 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:45.863616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] PATH_INFO: `/v3/projects/e944162cff6b4331bb86199026e95a0a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:45.864832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e944162cff6b4331bb86199026e95a0a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:45.867820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:45.876050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:45.877209 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:45.893759 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d3306-513c-40a5-b017-a5bf18056b24 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:45.899405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:45.900258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:45.900605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:45.900887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] PATH_INFO: `/v3/projects/2b2191df859c4ee29be70083c7ac50c6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:45.902903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b2191df859c4ee29be70083c7ac50c6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:45.904636 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:46.002200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e944162cff6b4331bb86199026e95a0a', 'request_id': 'req-6aa8caab-213c-4920-97e9-c6e96ab1941f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.048530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b2191df859c4ee29be70083c7ac50c6', 'request_id': 'req-a415cd12-556c-4051-9b84-3f0e2c40f8cb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.054705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b2191df859c4ee29be70083c7ac50c6', 'request_id': 'req-a415cd12-556c-4051-9b84-3f0e2c40f8cb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.061087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b2191df859c4ee29be70083c7ac50c6', 'request_id': 'req-a415cd12-556c-4051-9b84-3f0e2c40f8cb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.087442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e944162cff6b4331bb86199026e95a0a', 'request_id': 'req-6aa8caab-213c-4920-97e9-c6e96ab1941f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.089723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:46.094455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e944162cff6b4331bb86199026e95a0a', 'request_id': 'req-6aa8caab-213c-4920-97e9-c6e96ab1941f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.105928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] The token cache is being invalidate because project e944162cff6b4331bb86199026e95a0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:46.106202 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.106834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.108303 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1116/2243] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:45 2022] DELETE /identity/v3/projects/e944162cff6b4331bb86199026e95a0a => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:46.112071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:46.113782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:46.167868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] The token cache is being invalidate because project 2b2191df859c4ee29be70083c7ac50c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:46.168269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.168903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.170439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1128/2244] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:45 2022] DELETE /identity/v3/projects/2b2191df859c4ee29be70083c7ac50c6 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:46.174403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:46.175940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:46.181121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa8caab-213c-4920-97e9-c6e96ab1941f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:46.237582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:46.238480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:46.238765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:46.239039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] PATH_INFO: `/v3/users/403ef609de9f469ba82940d1803bf1c6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:46.240220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] RBAC: Authorizing `identity:delete_user(user_id=403ef609de9f469ba82940d1803bf1c6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:46.242412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:46.245072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415cd12-556c-4051-9b84-3f0e2c40f8cb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:46.249821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:46.250454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:46.250742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:46.251022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] PATH_INFO: `/v3/users/61858605a1d74b9c8efc4fe7c54a20dd` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:46.252168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=61858605a1d74b9c8efc4fe7c54a20dd)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:46.254145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:46.455995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '403ef609de9f469ba82940d1803bf1c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.469826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '403ef609de9f469ba82940d1803bf1c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.486529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:46.489515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61858605a1d74b9c8efc4fe7c54a20dd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.490186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61858605a1d74b9c8efc4fe7c54a20dd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.502085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61858605a1d74b9c8efc4fe7c54a20dd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.509571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61858605a1d74b9c8efc4fe7c54a20dd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.510223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61858605a1d74b9c8efc4fe7c54a20dd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.539022 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '403ef609de9f469ba82940d1803bf1c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.539652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '403ef609de9f469ba82940d1803bf1c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.540273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '403ef609de9f469ba82940d1803bf1c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:46.553137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1117/2245] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:46 2022] DELETE /identity/v3/users/403ef609de9f469ba82940d1803bf1c6 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:46.606989 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1129/2246] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:46 2022] DELETE /identity/v3/users/61858605a1d74b9c8efc4fe7c54a20dd => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:47.054921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:47.056432 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:47.073698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:47.096029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:47.097864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:47.132931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cca50e1-50ec-4d76-9930-162e3a3a0665 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:47.175666 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fc446f-c4bc-42eb-9e9e-f023c1788ef0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:47.181022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:47.182136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.182511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.182861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] PATH_INFO: `/v3/projects/9cb5688269aa41548705c8e7d6f8396f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.184075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cb5688269aa41548705c8e7d6f8396f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:47.186255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:47.190644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:47.191267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.191552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.191830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] PATH_INFO: `/v3/projects/441563a8d2944235b43b77009b434105` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.192950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] RBAC: Authorizing `identity:delete_project(project_id=441563a8d2944235b43b77009b434105)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:47.195719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:47.310784 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cb5688269aa41548705c8e7d6f8396f', 'request_id': 'req-73c81aa4-35fc-40d8-9823-e8b56616eedd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.316862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cb5688269aa41548705c8e7d6f8396f', 'request_id': 'req-73c81aa4-35fc-40d8-9823-e8b56616eedd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.321809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '441563a8d2944235b43b77009b434105', 'request_id': 'req-a4eda0d4-e012-4e3c-a929-9e3570e4014d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.323628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cb5688269aa41548705c8e7d6f8396f', 'request_id': 'req-73c81aa4-35fc-40d8-9823-e8b56616eedd'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.389933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '441563a8d2944235b43b77009b434105', 'request_id': 'req-a4eda0d4-e012-4e3c-a929-9e3570e4014d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.397212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '441563a8d2944235b43b77009b434105', 'request_id': 'req-a4eda0d4-e012-4e3c-a929-9e3570e4014d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.408689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] The token cache is being invalidate because project 441563a8d2944235b43b77009b434105 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:47.408984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.409638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a4eda0d4-e012-4e3c-a929-9e3570e4014d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.411126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1118/2247] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:47 2022] DELETE /identity/v3/projects/441563a8d2944235b43b77009b434105 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:47.416777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb79994f-d54a-454c-a7a2-35d9efcb26da None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.417174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb79994f-d54a-454c-a7a2-35d9efcb26da None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.417420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb79994f-d54a-454c-a7a2-35d9efcb26da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.436936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bb79994f-d54a-454c-a7a2-35d9efcb26da None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:47.438166 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bb79994f-d54a-454c-a7a2-35d9efcb26da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:47.452302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] The token cache is being invalidate because project 9cb5688269aa41548705c8e7d6f8396f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:47.452568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.456368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.456368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1130/2248] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:47 2022] DELETE /identity/v3/projects/9cb5688269aa41548705c8e7d6f8396f => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:47.458642 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:47.460645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:47.486345 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:47.513896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1119/2249] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:39:47.515547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511086d-b5ad-4292-ac7e-68da168ae7ef None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.516829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511086d-b5ad-4292-ac7e-68da168ae7ef None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.517243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4511086d-b5ad-4292-ac7e-68da168ae7ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.529089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:47.549503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4511086d-b5ad-4292-ac7e-68da168ae7ef None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:47.549503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4511086d-b5ad-4292-ac7e-68da168ae7ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:47.549696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c81aa4-35fc-40d8-9823-e8b56616eedd admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:47.549981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:47.550651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.550935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.551213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] PATH_INFO: `/v3/users/e0035f9c629148d0a7b48b29bfe8f049` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.552292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1120/2250] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:47.552606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0035f9c629148d0a7b48b29bfe8f049)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:47.553177 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4511086d-b5ad-4292-ac7e-68da168ae7ef None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:47.554599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:47.568544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4511086d-b5ad-4292-ac7e-68da168ae7ef None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:47.568544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:47.568544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.568544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.568984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.569088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:39:47.569356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:47.571603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:47.583251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1121/2251] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:39:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:39:47.588138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:47.598007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783862dc-1aa0-4483-86a5-36e455018cb1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:47.603517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:47.604109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.604394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.604673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.605292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:47.607205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:47.711999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1122/2252] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:47 2022] POST /identity/v3/projects => generated 371 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:47.716414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:47.768345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0035f9c629148d0a7b48b29bfe8f049'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.768994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0035f9c629148d0a7b48b29bfe8f049'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.780844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da65f451-733d-438c-a171-164447f2a52a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:47.781734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0035f9c629148d0a7b48b29bfe8f049'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.785889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:47.786506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:47.786790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:47.787072 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:47.787708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:47.792402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:47.792514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0035f9c629148d0a7b48b29bfe8f049'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.792514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0035f9c629148d0a7b48b29bfe8f049'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:47.915975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1123/2253] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:47 2022] POST /identity/v3/users => generated 286 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:39:47.916264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1131/2254] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:39:47 2022] DELETE /identity/v3/users/e0035f9c629148d0a7b48b29bfe8f049 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:47.924093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:47.926181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:47.947483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be47b83-100e-4088-be67-d073d291ab89 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:48.022483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:48.023286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.023866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.024173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.024909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:48.026632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:48.037298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1124/2255] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:47 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:48.041443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:48.051283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4f39d-a970-4f45-8a44-d11890c0cbd2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:48.056632 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e91559-d44a-446e-ae2c-2a663ba371d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:48.057234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e91559-d44a-446e-ae2c-2a663ba371d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.057541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e91559-d44a-446e-ae2c-2a663ba371d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.057812 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e91559-d44a-446e-ae2c-2a663ba371d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.058534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e91559-d44a-446e-ae2c-2a663ba371d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:48.060113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e91559-d44a-446e-ae2c-2a663ba371d5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:48.071732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1132/2256] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:48.076080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:48.086573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817b6e2e-6545-4b00-a4a9-bc0395c7fb9c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:48.091162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:48.092997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.093288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.093624 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] PATH_INFO: `/v3/projects/1efb2fdb645d42a7afef5dd341238938/users/0607119f0535468892ed1a271f8ecca1/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.111161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1efb2fdb645d42a7afef5dd341238938, user_id=0607119f0535468892ed1a271f8ecca1, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:48.114308 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:48.207823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1125/2257] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:48 2022] PUT /identity/v3/projects/1efb2fdb645d42a7afef5dd341238938/users/0607119f0535468892ed1a271f8ecca1/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:48.213766 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.214247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.214342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.236545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:48.238366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:48.308321 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1133/2258] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:48.318430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:48.320271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:48.333819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:48.360294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:48.362404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:48.378312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3152e3-3314-45b5-9689-e7866637b9ad None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:48.384900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:48.386340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.386575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54b3beb-e284-4e7f-90b5-859eed5a8f94 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:48.386748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.387168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] PATH_INFO: `/v3/projects/a9883b2b496a47f4a66d4adca797c108` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.388380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9883b2b496a47f4a66d4adca797c108)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:48.390427 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:48.397256 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cac807-ae49-40bd-bf1b-f366c27cb1ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:48.398488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cac807-ae49-40bd-bf1b-f366c27cb1ed service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.398964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cac807-ae49-40bd-bf1b-f366c27cb1ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.399474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cac807-ae49-40bd-bf1b-f366c27cb1ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.420264 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cac807-ae49-40bd-bf1b-f366c27cb1ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:48.422080 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cac807-ae49-40bd-bf1b-f366c27cb1ed service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:48.433304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1126/2259] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:48.521769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9883b2b496a47f4a66d4adca797c108', 'request_id': 'req-d430850f-359b-4000-ba27-0fabc91c5dae'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:48.526304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:48.528252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9883b2b496a47f4a66d4adca797c108', 'request_id': 'req-d430850f-359b-4000-ba27-0fabc91c5dae'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:48.538599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9883b2b496a47f4a66d4adca797c108', 'request_id': 'req-d430850f-359b-4000-ba27-0fabc91c5dae'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:48.629314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] The token cache is being invalidate because project a9883b2b496a47f4a66d4adca797c108 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:39:48.629729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:48.630406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-d430850f-359b-4000-ba27-0fabc91c5dae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:39:48.632664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1134/2260] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:39:48 2022] DELETE /identity/v3/projects/a9883b2b496a47f4a66d4adca797c108 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:39:48.638517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3269958-5884-4b75-b312-f1704bc0252b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.638803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3269958-5884-4b75-b312-f1704bc0252b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.639077 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3269958-5884-4b75-b312-f1704bc0252b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.659349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e3269958-5884-4b75-b312-f1704bc0252b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:48.660537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e3269958-5884-4b75-b312-f1704bc0252b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:48.718490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1127/2261] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:48.726708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837f89ea-1dd8-4ac0-b930-5f4b3726ea75 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.727002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837f89ea-1dd8-4ac0-b930-5f4b3726ea75 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.727285 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837f89ea-1dd8-4ac0-b930-5f4b3726ea75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.747824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-837f89ea-1dd8-4ac0-b930-5f4b3726ea75 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:48.750082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-837f89ea-1dd8-4ac0-b930-5f4b3726ea75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:48.761288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1135/2262] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:48.765715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3269958-5884-4b75-b312-f1704bc0252b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:48.776888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3269958-5884-4b75-b312-f1704bc0252b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:48.782235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:48.782859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.783145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.783513 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.784074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:39:48.784382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:48.786069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:48.799964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1128/2263] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:39:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:39:48.805212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837f89ea-1dd8-4ac0-b930-5f4b3726ea75 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:48.815787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837f89ea-1dd8-4ac0-b930-5f4b3726ea75 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:48.821725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:48.822459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:48.822764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:48.823204 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:48.823845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:48.825825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:48.941725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1136/2264] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:48 2022] POST /identity/v3/projects => generated 377 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:48.946261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:48.956594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e952f9-c035-449d-b56d-aa6d06d911b6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:49.022454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:49.023775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:49.024157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:49.024530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:49.025179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:49.027154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:49.161923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1129/2265] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:48 2022] POST /identity/v3/users => generated 289 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:49.166747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:49.176980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6814cc0a-11e7-48c5-a398-33becc38e63a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:49.181789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:49.182390 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:49.182921 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:49.183215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:49.184016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:49.186871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:49.200780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1137/2266] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:49.202506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:49.217788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487e478f-043d-4f9a-971a-0d2228cffc70 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:49.224367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb02976-3029-4b87-b60b-02e895df413b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:49.225006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb02976-3029-4b87-b60b-02e895df413b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:49.225296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb02976-3029-4b87-b60b-02e895df413b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:49.225708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb02976-3029-4b87-b60b-02e895df413b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:49.231191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb02976-3029-4b87-b60b-02e895df413b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:49.231191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb02976-3029-4b87-b60b-02e895df413b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:49.244575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1130/2267] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:49 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:49.246221 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:49.257818 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da284255-fc0d-4cf7-9c37-6c4a8e47aa32 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:49.262583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:49.263815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:49.264104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:49.264381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] PATH_INFO: `/v3/projects/ab39b3f3a2914a0d8a53016e893ab460/users/590d90833ba648b7b1beb0c67f953f5b/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:49.281024 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab39b3f3a2914a0d8a53016e893ab460, user_id=590d90833ba648b7b1beb0c67f953f5b, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:49.283753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:49.464199 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1138/2268] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:49 2022] PUT /identity/v3/projects/ab39b3f3a2914a0d8a53016e893ab460/users/590d90833ba648b7b1beb0c67f953f5b/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:49.476378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d924d10e-ab07-422d-970e-bf48ed0c64ba None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:49.476665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d924d10e-ab07-422d-970e-bf48ed0c64ba None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:49.476952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d924d10e-ab07-422d-970e-bf48ed0c64ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:49.507625 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d924d10e-ab07-422d-970e-bf48ed0c64ba None None] MFA Rules not processed for user `590d90833ba648b7b1beb0c67f953f5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:49.509201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d924d10e-ab07-422d-970e-bf48ed0c64ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:49.518076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:49.583808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1131/2269] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:49.595581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:49.597342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:49.690780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bfda8f-127a-49f0-af0b-d53e7f7d2f5f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:49.695256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7fae443ad92439aadd2f94ad3bca594', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:49.695943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:49.696230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:49.696503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:49.704886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:49.706395 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:49.715643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1139/2270] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:49 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:39:49.894702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3100407-2756-4a8c-9c8d-7a2cd843d54d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:49.894702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3100407-2756-4a8c-9c8d-7a2cd843d54d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:49.894702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3100407-2756-4a8c-9c8d-7a2cd843d54d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:49.927618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f3100407-2756-4a8c-9c8d-7a2cd843d54d None None] MFA Rules not processed for user `0607119f0535468892ed1a271f8ecca1`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:49.927618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f3100407-2756-4a8c-9c8d-7a2cd843d54d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:50.044578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1132/2271] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:49 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:50.049985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:50.051701 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:50.118601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358d5d99-efbd-4537-9a8b-14bed6d1f84d service glance] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:50.123147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:50.123753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:50.124035 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:50.124310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:50.124946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:50.126885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:50.132728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:50.224868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1140/2272] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:39:50 2022] POST /identity/v3/projects => generated 373 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:50.229337 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3100407-2756-4a8c-9c8d-7a2cd843d54d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:50.240209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3100407-2756-4a8c-9c8d-7a2cd843d54d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:50.298950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:50.299569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:50.299857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:50.300139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:50.300772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:50.302742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:50.413184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1133/2273] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:39:50 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:39:50.418363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:50.429558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe331b1-cbcc-43c1-8c95-b4cd26443a6f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:50.438190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:50.438801 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:50.439085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:50.439357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:50.439994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:50.441647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:50.453341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1141/2274] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:50.457556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:50.467871 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba93b52-9800-4a1d-9fc9-280e313a0aed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:50.472961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:50.473387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:50.473679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:50.473957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] PATH_INFO: `/v3/projects/7065e65475944d70a4c757caccd2563f/users/2c770d58498f43d5800b235b9f1ce91e/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:50.489222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7065e65475944d70a4c757caccd2563f, user_id=2c770d58498f43d5800b235b9f1ce91e, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:50.492542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:50.587858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1134/2275] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:50 2022] PUT /identity/v3/projects/7065e65475944d70a4c757caccd2563f/users/2c770d58498f43d5800b235b9f1ce91e/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:50.591254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:50.602305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47548d16-ff38-4390-8e2c-7df774d463f1 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:50.662323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c57d612-c887-4a62-a5ad-d610ca2d3df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:50.662988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c57d612-c887-4a62-a5ad-d610ca2d3df7 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:50.663930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c57d612-c887-4a62-a5ad-d610ca2d3df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:50.664436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c57d612-c887-4a62-a5ad-d610ca2d3df7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:50.665154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c57d612-c887-4a62-a5ad-d610ca2d3df7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:50.666829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c57d612-c887-4a62-a5ad-d610ca2d3df7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:50.677648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1142/2276] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:39:50 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:39:50.682522 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:50.692689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:39:50.698636 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0750ee63-99e8-4d93-9a05-7e715aaea108 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:50.704455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:50.705497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:50.705791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:50.713618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] PATH_INFO: `/v3/projects/7065e65475944d70a4c757caccd2563f/users/2c770d58498f43d5800b235b9f1ce91e/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:50.713618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7065e65475944d70a4c757caccd2563f, user_id=2c770d58498f43d5800b235b9f1ce91e, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:50.713618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:50.721792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1135/2277] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:39:50 2022] PUT /identity/v3/projects/7065e65475944d70a4c757caccd2563f/users/2c770d58498f43d5800b235b9f1ce91e/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:39:52.231562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2670a7d7-5333-47d1-880f-14dd3203c928 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:52.232583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2670a7d7-5333-47d1-880f-14dd3203c928 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:52.232719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2670a7d7-5333-47d1-880f-14dd3203c928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:52.270359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2670a7d7-5333-47d1-880f-14dd3203c928 None None] MFA Rules not processed for user `2c770d58498f43d5800b235b9f1ce91e`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:39:52.271706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2670a7d7-5333-47d1-880f-14dd3203c928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:52.369282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1143/2278] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:39:52 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:39:52.382927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:52.384684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:39:52.446758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60adaba6-0757-45b1-92f2-f0d737570c75 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:52.450594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:52.451381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:52.451669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:52.451947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:52.510479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:52.511926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:52.521231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1136/2279] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:52 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:39:54.423688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2670a7d7-5333-47d1-880f-14dd3203c928 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:39:54.433674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2670a7d7-5333-47d1-880f-14dd3203c928 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:39:54.437131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:39:54.437775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:39:54.438056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:39:54.438325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:39:54.447842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:39:54.449988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:39:54.462435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1144/2280] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:39:54 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:00.189158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:00.190811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:00.268899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613fc071-58c4-4228-8331-d2adc595b9ab service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:00.274407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f167be-aa9b-451f-a67d-fda40c5b05ae service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:00.275032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f167be-aa9b-451f-a67d-fda40c5b05ae service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:00.275415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f167be-aa9b-451f-a67d-fda40c5b05ae service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:00.275879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f167be-aa9b-451f-a67d-fda40c5b05ae service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:00.278057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f6f167be-aa9b-451f-a67d-fda40c5b05ae service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:00.293594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f167be-aa9b-451f-a67d-fda40c5b05ae service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:00.293594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f167be-aa9b-451f-a67d-fda40c5b05ae service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:00.303777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1137/2281] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:04.455236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=79,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:10.299159 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=83,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:13.665428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:13.729176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf2e912-9015-42b8-b686-ce530f964da7 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:13.734614 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:13.735250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:13.735537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:13.735817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] PATH_INFO: `/v3/users/590d90833ba648b7b1beb0c67f953f5b` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:13.737007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=590d90833ba648b7b1beb0c67f953f5b)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:13.739178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:13.895363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '590d90833ba648b7b1beb0c67f953f5b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:13.896065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '590d90833ba648b7b1beb0c67f953f5b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:13.908721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '590d90833ba648b7b1beb0c67f953f5b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:13.915748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '590d90833ba648b7b1beb0c67f953f5b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:13.917789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '590d90833ba648b7b1beb0c67f953f5b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:14.024743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1145/2282] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:40:13 2022] DELETE /identity/v3/users/590d90833ba648b7b1beb0c67f953f5b => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:14.032711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:14.033254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:14.033385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:14.052743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:14.054750 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:14.114285 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1138/2283] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:14.123700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:14.125228 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:14.186095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad19423-ac6e-478f-b99e-0f8edbb40c1d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:14.190393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ac346a-26e8-4ead-be9d-f29e3c5869ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:14.191006 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac346a-26e8-4ead-be9d-f29e3c5869ed service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:14.192003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac346a-26e8-4ead-be9d-f29e3c5869ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:14.192003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac346a-26e8-4ead-be9d-f29e3c5869ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:14.202215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ac346a-26e8-4ead-be9d-f29e3c5869ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:14.204407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ac346a-26e8-4ead-be9d-f29e3c5869ed service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:14.214725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1146/2284] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:14.619520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:14.620969 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:14.633984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d38402-1e2d-4889-83d6-82134757c9d3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:14.639725 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:14.640579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:14.640661 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:14.640956 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] PATH_INFO: `/v3/projects/ab39b3f3a2914a0d8a53016e893ab460` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:14.642060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab39b3f3a2914a0d8a53016e893ab460)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:14.644160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:14.798441 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab39b3f3a2914a0d8a53016e893ab460', 'request_id': 'req-2ed67601-652b-4771-9e78-6aedff36072c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:14.887020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab39b3f3a2914a0d8a53016e893ab460', 'request_id': 'req-2ed67601-652b-4771-9e78-6aedff36072c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:14.894081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab39b3f3a2914a0d8a53016e893ab460', 'request_id': 'req-2ed67601-652b-4771-9e78-6aedff36072c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:14.904463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] The token cache is being invalidate because project ab39b3f3a2914a0d8a53016e893ab460 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:40:14.904753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:14.906119 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2ed67601-652b-4771-9e78-6aedff36072c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:14.907696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1139/2285] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:40:14 2022] DELETE /identity/v3/projects/ab39b3f3a2914a0d8a53016e893ab460 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:14.912486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22666a44-8f12-4061-803c-70e530d2041d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:14.912969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22666a44-8f12-4061-803c-70e530d2041d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:14.913076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22666a44-8f12-4061-803c-70e530d2041d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:14.933163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-22666a44-8f12-4061-803c-70e530d2041d None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:14.934382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-22666a44-8f12-4061-803c-70e530d2041d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:14.951928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:15.000448 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1147/2286] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:15.008823 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db68a98-442f-4fb9-a2f6-727f09e316fc None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.009122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db68a98-442f-4fb9-a2f6-727f09e316fc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.009437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db68a98-442f-4fb9-a2f6-727f09e316fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.032712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4db68a98-442f-4fb9-a2f6-727f09e316fc None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:15.033924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4db68a98-442f-4fb9-a2f6-727f09e316fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:15.042162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1140/2287] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:15.047067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22666a44-8f12-4061-803c-70e530d2041d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:15.056662 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22666a44-8f12-4061-803c-70e530d2041d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:15.061144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:15.061815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.062100 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.062375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.063020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:40:15.063324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:15.064951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:15.076409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1148/2288] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:40:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:40:15.081029 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db68a98-442f-4fb9-a2f6-727f09e316fc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:15.090613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db68a98-442f-4fb9-a2f6-727f09e316fc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:15.096035 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:15.096649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.096933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.097210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.097888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:15.099889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:15.190133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1141/2289] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:40:15 2022] POST /identity/v3/projects => generated 373 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:15.194157 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:15.205103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d52e78-4a86-4338-8d52-d0beacc0eb0d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:15.263920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:15.265054 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.265368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.265646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.266270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:15.268498 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:15.348256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1149/2290] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:40:15 2022] POST /identity/v3/users => generated 287 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:15.349271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:15.359955 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf07ed2-ae96-47e2-8d25-484a11eb0e60 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:15.365105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:15.365949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.366301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.366586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.367311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:15.369209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:15.380995 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1142/2291] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:40:15.385014 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:15.396919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e64ef5-ac0f-44d1-bb79-7befac2ce3d2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:15.401775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fca77c-205f-4fe4-a586-6d350de32d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:15.402382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fca77c-205f-4fe4-a586-6d350de32d68 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.402669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fca77c-205f-4fe4-a586-6d350de32d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.402949 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fca77c-205f-4fe4-a586-6d350de32d68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.403592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fca77c-205f-4fe4-a586-6d350de32d68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:15.405867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fca77c-205f-4fe4-a586-6d350de32d68 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:15.416623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1150/2292] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:40:15.420794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:15.430737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b878938-333e-4d09-9ae6-5e4365f27692 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:15.435413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:15.436064 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.437010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.437010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] PATH_INFO: `/v3/projects/40b9acd32a1f4bbcb3bf71ac93bae1f5/users/fef6a70cbb564a55917ae1d9ae2f71e1/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.456454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40b9acd32a1f4bbcb3bf71ac93bae1f5, user_id=fef6a70cbb564a55917ae1d9ae2f71e1, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:15.459407 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:15.554966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1143/2293] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:40:15 2022] PUT /identity/v3/projects/40b9acd32a1f4bbcb3bf71ac93bae1f5/users/fef6a70cbb564a55917ae1d9ae2f71e1/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:15.560389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96641491-cbad-436c-9994-48603ea27394 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.560601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96641491-cbad-436c-9994-48603ea27394 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.560882 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96641491-cbad-436c-9994-48603ea27394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.582218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-96641491-cbad-436c-9994-48603ea27394 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:15.583481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-96641491-cbad-436c-9994-48603ea27394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:15.647988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1151/2294] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:15.658062 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:15.659917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:15.683214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:15.722459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ce58f1-4153-4e76-82d5-f9805fffc470 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:15.726248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:15.726866 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:15.727148 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:15.727420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:15.737216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:15.738997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:15.750800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1144/2295] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:16.172244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96641491-cbad-436c-9994-48603ea27394 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:16.173722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-96641491-cbad-436c-9994-48603ea27394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:16.211454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:16.250544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96641491-cbad-436c-9994-48603ea27394 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:16.267477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046c91a8-27e5-4364-9451-1d2cbcdea0f6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:16.267477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c91a8-27e5-4364-9451-1d2cbcdea0f6 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:16.267477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c91a8-27e5-4364-9451-1d2cbcdea0f6 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:16.267477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c91a8-27e5-4364-9451-1d2cbcdea0f6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:16.267817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-046c91a8-27e5-4364-9451-1d2cbcdea0f6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:16.275050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046c91a8-27e5-4364-9451-1d2cbcdea0f6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:16.276880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046c91a8-27e5-4364-9451-1d2cbcdea0f6 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:16.301075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1152/2296] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:16.980011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:16.981834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:17.004462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8231599c-269a-4bc4-9c4f-781d84517774 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:17.004684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:17.004939 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:17.005153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:17.009429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:17.010175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:17.010536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:17.010883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] PATH_INFO: `/v3/users/0607119f0535468892ed1a271f8ecca1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:17.012395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0607119f0535468892ed1a271f8ecca1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:17.015252 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:17.040116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] MFA Rules not processed for user `fef6a70cbb564a55917ae1d9ae2f71e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:17.041413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:17.129530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1153/2297] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:17.141404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:17.142913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:17.175746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0607119f0535468892ed1a271f8ecca1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.190899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0607119f0535468892ed1a271f8ecca1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.207606 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc816534-8a5f-4ac2-8523-59a53ee9fa5f None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:17.213972 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:17.214698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:17.214983 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:17.215259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:17.224539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:17.225981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:17.234879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1154/2298] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:40:17.264031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0607119f0535468892ed1a271f8ecca1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.265388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0607119f0535468892ed1a271f8ecca1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.267442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0607119f0535468892ed1a271f8ecca1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.279384 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1145/2299] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:40:16 2022] DELETE /identity/v3/users/0607119f0535468892ed1a271f8ecca1 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:17.557161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:17.569091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dff36c-6c55-4205-b2cb-08b28041a829 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:17.619380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:17.629724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:17.630382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:17.630684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:17.630955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] PATH_INFO: `/v3/projects/1efb2fdb645d42a7afef5dd341238938` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:17.632075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=1efb2fdb645d42a7afef5dd341238938)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:17.634008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:17.739556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1efb2fdb645d42a7afef5dd341238938', 'request_id': 'req-f3521fae-8503-4980-bc51-075ddb42a6ae'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.745366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1efb2fdb645d42a7afef5dd341238938', 'request_id': 'req-f3521fae-8503-4980-bc51-075ddb42a6ae'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.752255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1efb2fdb645d42a7afef5dd341238938', 'request_id': 'req-f3521fae-8503-4980-bc51-075ddb42a6ae'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.801467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] The token cache is being invalidate because project 1efb2fdb645d42a7afef5dd341238938 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:40:17.801777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.802413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:17.804600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1155/2300] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:40:17 2022] DELETE /identity/v3/projects/1efb2fdb645d42a7afef5dd341238938 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:17.808261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:17.809756 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:17.823645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d0c497-9c23-42d9-ab09-db22b7003e41 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:17.880754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:17.881418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:17.881706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:17.881983 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] PATH_INFO: `/v3/users/2c770d58498f43d5800b235b9f1ce91e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:17.883134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c770d58498f43d5800b235b9f1ce91e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:17.885107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:18.132909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:18.141822 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c770d58498f43d5800b235b9f1ce91e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.153924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c770d58498f43d5800b235b9f1ce91e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.206999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c770d58498f43d5800b235b9f1ce91e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.207654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c770d58498f43d5800b235b9f1ce91e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.208271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-68c22806-6757-4715-a99c-cef6ed1ff2bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c770d58498f43d5800b235b9f1ce91e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.221679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1146/2301] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:40:17 2022] DELETE /identity/v3/users/2c770d58498f43d5800b235b9f1ce91e => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:18.472424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:18.473922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:18.503513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3521fae-8503-4980-bc51-075ddb42a6ae admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:18.568182 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:18.568809 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:18.569091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:18.569388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] PATH_INFO: `/v3/projects/7065e65475944d70a4c757caccd2563f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:18.570525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7065e65475944d70a4c757caccd2563f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:18.572496 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:18.679017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7065e65475944d70a4c757caccd2563f', 'request_id': 'req-529edb90-890d-46bf-95d3-0804e5aea4f1'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.684511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7065e65475944d70a4c757caccd2563f', 'request_id': 'req-529edb90-890d-46bf-95d3-0804e5aea4f1'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.693569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7065e65475944d70a4c757caccd2563f', 'request_id': 'req-529edb90-890d-46bf-95d3-0804e5aea4f1'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.776735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] The token cache is being invalidate because project 7065e65475944d70a4c757caccd2563f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:40:18.777023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.777651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-529edb90-890d-46bf-95d3-0804e5aea4f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:18.779309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1156/2302] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:40:18 2022] DELETE /identity/v3/projects/7065e65475944d70a4c757caccd2563f => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:18.785010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428b316-8032-485f-8113-179736f2b5cf None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:18.785298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428b316-8032-485f-8113-179736f2b5cf None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:18.785601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428b316-8032-485f-8113-179736f2b5cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:18.806009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8428b316-8032-485f-8113-179736f2b5cf None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:18.807200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8428b316-8032-485f-8113-179736f2b5cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:18.874918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1147/2303] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:40:18.878137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d92895-aab9-4343-91b9-a075d4dfcedf None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:18.882340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d92895-aab9-4343-91b9-a075d4dfcedf None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:18.882340 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d92895-aab9-4343-91b9-a075d4dfcedf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:18.898767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-08d92895-aab9-4343-91b9-a075d4dfcedf None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:18.899951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-08d92895-aab9-4343-91b9-a075d4dfcedf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:18.908767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1157/2304] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:18.913137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8428b316-8032-485f-8113-179736f2b5cf None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:18.922675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27445bc-f99a-40e1-b718-7fdd7bfffd90 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:18.922987 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27445bc-f99a-40e1-b718-7fdd7bfffd90 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:18.923264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27445bc-f99a-40e1-b718-7fdd7bfffd90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:18.923647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8428b316-8032-485f-8113-179736f2b5cf None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:18.928392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:18.929011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:18.929297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:18.929609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:18.930305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:40:18.930615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:18.931341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:18.932365 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:18.942825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c27445bc-f99a-40e1-b718-7fdd7bfffd90 None None] MFA Rules not processed for user `fef6a70cbb564a55917ae1d9ae2f71e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:18.944488 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c27445bc-f99a-40e1-b718-7fdd7bfffd90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:18.944881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1148/2305] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:40:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:40:18.949912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:18.961173 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e912bfb5-39be-4e16-9fa5-e3761fb544ef admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:18.966468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:18.967082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:18.969864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:18.969864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:18.969864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:18.971433 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.006152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1158/2306] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:40:19.095045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1149/2307] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:40:18 2022] POST /identity/v3/projects => generated 383 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:19.096213 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27445bc-f99a-40e1-b718-7fdd7bfffd90 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:19.107646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27445bc-f99a-40e1-b718-7fdd7bfffd90 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:19.166518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:19.167144 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.167431 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.167707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.168342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:19.170327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.248532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:19.254354 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:19.254482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1159/2308] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:40:19 2022] POST /identity/v3/users => generated 292 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:19.257789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:19.269663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3bf144-f210-43b5-be7c-ab9a45b5d576 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:19.274547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:19.275143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.275426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.275696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.276392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:19.278158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.288799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1160/2309] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:40:19.292769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:19.303006 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20bf53b-90fc-4cca-8544-9aa0d10ae7f8 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:19.307699 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:19.308289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.308570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.308853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.309518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:19.311099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.315327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdcc942-523f-4fb5-b7e2-d64b9fd25625 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:19.319066 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecbb173-e8b4-4e15-8475-fe68a1890dbb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:19.319678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbb173-e8b4-4e15-8475-fe68a1890dbb service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.319961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbb173-e8b4-4e15-8475-fe68a1890dbb service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.324768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbb173-e8b4-4e15-8475-fe68a1890dbb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.324768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7ecbb173-e8b4-4e15-8475-fe68a1890dbb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:19.324973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1161/2310] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:40:19.326772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:19.332797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecbb173-e8b4-4e15-8475-fe68a1890dbb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:19.334265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecbb173-e8b4-4e15-8475-fe68a1890dbb service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.338241 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417d5fca-6a7d-4366-b439-c6e70cf6e87d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:19.343002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1150/2311] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:40:19.343155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:19.343639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.343914 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.344189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] PATH_INFO: `/v3/projects/39e8dfe031dc4da7a16c6014f683e627/users/1142d077012246239795a4d006eb2971/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.359416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39e8dfe031dc4da7a16c6014f683e627, user_id=1142d077012246239795a4d006eb2971, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:19.362179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.428835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1162/2312] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:40:19 2022] PUT /identity/v3/projects/39e8dfe031dc4da7a16c6014f683e627/users/1142d077012246239795a4d006eb2971/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:40:19.440664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0e4c7-9bfe-44fa-8d39-fb974a130f91 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.440992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0e4c7-9bfe-44fa-8d39-fb974a130f91 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.441304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0e4c7-9bfe-44fa-8d39-fb974a130f91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.473551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b2d0e4c7-9bfe-44fa-8d39-fb974a130f91 None None] MFA Rules not processed for user `1142d077012246239795a4d006eb2971`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:19.475839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b2d0e4c7-9bfe-44fa-8d39-fb974a130f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:19.486187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:19.569511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1151/2313] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:19.570783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:19.637513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:19.650268 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03361e23-3641-49d2-bac0-e47f61887546 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:19.655651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:19.656408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.656763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.657103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.662188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:19.662188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.794454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1163/2314] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:40:19 2022] POST /identity/v3/projects => generated 385 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:19.799047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0e4c7-9bfe-44fa-8d39-fb974a130f91 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:19.809833 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0e4c7-9bfe-44fa-8d39-fb974a130f91 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:19.868137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:19.868808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:19.869165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:19.869475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:19.870132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:19.872198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:19.990896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1152/2315] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:40:19 2022] POST /identity/v3/users => generated 299 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:40:19.995136 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:20.006263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b7f2af-a210-4647-930b-2bcaa14b854a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:20.010964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:20.011609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:20.011894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:20.014820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:20.014820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:20.014820 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:20.025476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1164/2316] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:40:20.030316 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:20.040278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17554c1a-c3f8-4197-bf89-c06173829ae5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:20.044965 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:20.046193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:20.046447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:20.046723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] PATH_INFO: `/v3/projects/e6b7311400ac4969825d2e5fa9898f76/users/79bd2ed4d4e949499525551cdf0e8466/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:20.062172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6b7311400ac4969825d2e5fa9898f76, user_id=79bd2ed4d4e949499525551cdf0e8466, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:20.064825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:20.143547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1153/2317] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:40:20 2022] PUT /identity/v3/projects/e6b7311400ac4969825d2e5fa9898f76/users/79bd2ed4d4e949499525551cdf0e8466/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:20.147513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:20.157932 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4757ab-2d3a-437e-ace0-cdde6a833f5f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:20.214966 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c4b082-d2e5-44a4-b58b-879af9e4d3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:20.215572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c4b082-d2e5-44a4-b58b-879af9e4d3cd admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:20.215857 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c4b082-d2e5-44a4-b58b-879af9e4d3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:20.216133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c4b082-d2e5-44a4-b58b-879af9e4d3cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:20.216764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c4b082-d2e5-44a4-b58b-879af9e4d3cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:20.218415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c4b082-d2e5-44a4-b58b-879af9e4d3cd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:20.228080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1165/2318] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:40:20.232257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:20.243004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb06f09-aceb-41bd-acf8-39a6050f2f00 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:20.247645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:20.248348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:20.248619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:20.249829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] PATH_INFO: `/v3/projects/e6b7311400ac4969825d2e5fa9898f76/users/79bd2ed4d4e949499525551cdf0e8466/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:20.251362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6b7311400ac4969825d2e5fa9898f76, user_id=79bd2ed4d4e949499525551cdf0e8466, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:20.254042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:20.264391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1154/2319] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:40:20 2022] PUT /identity/v3/projects/e6b7311400ac4969825d2e5fa9898f76/users/79bd2ed4d4e949499525551cdf0e8466/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:20.275129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:20.275413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:20.275692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:20.303264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] MFA Rules not processed for user `79bd2ed4d4e949499525551cdf0e8466`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:20.304462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:20.372184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1166/2320] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:20 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:20.383354 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:20.385047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:20.441740 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc4374-a9c9-4fd3-bca0-f9e7e03e163a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:20.450436 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:20.451059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:20.451335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:20.451607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:20.469683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:20.509079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:20.510526 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:20.520936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1155/2321] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:20.819793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:20.821295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:20.886447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:20.906770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731dd7a9-d75d-4020-b17f-250b904fd99b None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:20.911757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:20.912353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:20.912638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:20.912916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:20.914675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:20.926750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:20.928997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:20.940036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1167/2322] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:25.038774 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:25.040283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:25.130840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6676f8b-73cb-42d1-99a1-58ed7fa76f18 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:25.136475 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:25.137094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:25.137410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:25.137692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:25.139487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:25.152452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:25.154107 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:25.165529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1156/2323] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:25.818164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:25.832566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:25.910568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb6ea4-d14b-4ecd-b8c6-7831c1a4cee4 service placement] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:25.915012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:25.915621 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:25.915899 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:25.916195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:25.918009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:25.986595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:25.987179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:25.988111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:25.988956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:26.001336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1168/2324] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:26.004500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f548da-0a65-4506-914d-86f5d454e4d6 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:26.008278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:26.008896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:26.009182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:26.009501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:26.030546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:26.030546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:26.041687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1157/2325] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:25 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:40:26.648615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:26.650202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:26.663098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52330d64-a821-4adb-8a3b-88b63367ba64 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:26.670172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:26.670786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:26.671123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:26.671408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] PATH_INFO: `/v3/users/14729da7649a4ab785ff87981968d342` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:26.672571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14729da7649a4ab785ff87981968d342)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:26.675110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:26.901704 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14729da7649a4ab785ff87981968d342'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:26.902343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14729da7649a4ab785ff87981968d342'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:26.915856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14729da7649a4ab785ff87981968d342'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:26.924771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14729da7649a4ab785ff87981968d342'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:26.925623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14729da7649a4ab785ff87981968d342'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:26.999735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:27.004672 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1169/2326] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:40:26 2022] DELETE /identity/v3/users/14729da7649a4ab785ff87981968d342 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:27.265688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:27.277963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0515d49-a02f-4d63-a89c-d6004f54d3c0 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:27.336921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:27.337515 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:27.337796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:27.338069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] PATH_INFO: `/v3/projects/62f0f20404ba41d3aa380a9d10eeab81` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:27.339171 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62f0f20404ba41d3aa380a9d10eeab81)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:27.341070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:27.364141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:27.462749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62f0f20404ba41d3aa380a9d10eeab81', 'request_id': 'req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:27.543637 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62f0f20404ba41d3aa380a9d10eeab81', 'request_id': 'req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:27.550830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62f0f20404ba41d3aa380a9d10eeab81', 'request_id': 'req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:27.561600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] The token cache is being invalidate because project 62f0f20404ba41d3aa380a9d10eeab81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:40:27.561903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:27.562546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:27.564155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1158/2327] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:40:27 2022] DELETE /identity/v3/projects/62f0f20404ba41d3aa380a9d10eeab81 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:31.606946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:31.608490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:31.687547 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a88655-3afa-4b0e-9436-845701156b40 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:31.770870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5148e29-6f67-4ba5-afe3-8156e5dc68a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:31.771502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5148e29-6f67-4ba5-afe3-8156e5dc68a1 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:31.771796 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5148e29-6f67-4ba5-afe3-8156e5dc68a1 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:31.772071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5148e29-6f67-4ba5-afe3-8156e5dc68a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:31.773931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d5148e29-6f67-4ba5-afe3-8156e5dc68a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:31.794726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5148e29-6f67-4ba5-afe3-8156e5dc68a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:31.796316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5148e29-6f67-4ba5-afe3-8156e5dc68a1 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:31.808310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1170/2328] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:31.921142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:31.923135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:31.986829 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07bbcfa-d5dc-4206-a0e9-b279b36385f2 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:31.991717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:31.992366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:31.992653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:31.992934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] PATH_INFO: `/v3/users/1142d077012246239795a4d006eb2971` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:31.994103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1142d077012246239795a4d006eb2971)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:31.996037 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:32.176878 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1142d077012246239795a4d006eb2971'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:32.189313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1142d077012246239795a4d006eb2971'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:32.257703 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1142d077012246239795a4d006eb2971'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:32.258549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1142d077012246239795a4d006eb2971'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:32.259178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1142d077012246239795a4d006eb2971'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:32.273273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1159/2329] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:40:31 2022] DELETE /identity/v3/users/1142d077012246239795a4d006eb2971 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:32.277787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:32.278075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:32.278355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:32.298339 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:32.299561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:32.359034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1171/2330] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:32.369528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:32.370964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:32.407329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:32.433150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e992819-4432-450c-b2b0-78776ce7171f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:32.436896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:32.438230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:32.438812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:32.439137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:32.448192 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:32.449653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:32.459538 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1160/2331] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:32.899380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:32.900851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:32.916813 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305c4c6-7695-4e2f-969b-862b8993a9f8 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:32.922727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:32.923681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:32.923971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:32.924254 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] PATH_INFO: `/v3/projects/39e8dfe031dc4da7a16c6014f683e627` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:32.925600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39e8dfe031dc4da7a16c6014f683e627)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:32.927616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:32.950377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:33.047318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39e8dfe031dc4da7a16c6014f683e627', 'request_id': 'req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.052817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39e8dfe031dc4da7a16c6014f683e627', 'request_id': 'req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.059165 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39e8dfe031dc4da7a16c6014f683e627', 'request_id': 'req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.166408 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] The token cache is being invalidate because project 39e8dfe031dc4da7a16c6014f683e627 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:40:33.166627 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.167230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.168736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1172/2332] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:40:32 2022] DELETE /identity/v3/projects/39e8dfe031dc4da7a16c6014f683e627 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:33.174062 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:33.184361 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b716f3b7-1097-413a-ba6f-3c7e90b94794 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:33.243819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:33.244455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:33.244738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:33.245016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] PATH_INFO: `/v3/users/79bd2ed4d4e949499525551cdf0e8466` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:33.246261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=79bd2ed4d4e949499525551cdf0e8466)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:33.248174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:33.432742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79bd2ed4d4e949499525551cdf0e8466'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.450940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79bd2ed4d4e949499525551cdf0e8466'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.533287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79bd2ed4d4e949499525551cdf0e8466'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.533944 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79bd2ed4d4e949499525551cdf0e8466'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.534546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-89368b2f-48c9-414f-bb7c-d07acc0b7a6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79bd2ed4d4e949499525551cdf0e8466'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:33.548099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1161/2333] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:40:33 2022] DELETE /identity/v3/users/79bd2ed4d4e949499525551cdf0e8466 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:33.991273 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:33.993931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:34.019342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb9ad4c-a542-4275-b66e-d292ed5d20c5 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:34.082659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:34.083302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.083588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.083863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] PATH_INFO: `/v3/projects/e6b7311400ac4969825d2e5fa9898f76` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.084992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6b7311400ac4969825d2e5fa9898f76)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:34.086971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:34.209430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6b7311400ac4969825d2e5fa9898f76', 'request_id': 'req-a9595dad-b078-435b-9331-56bb8a431044'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:34.215068 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6b7311400ac4969825d2e5fa9898f76', 'request_id': 'req-a9595dad-b078-435b-9331-56bb8a431044'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:34.222401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6b7311400ac4969825d2e5fa9898f76', 'request_id': 'req-a9595dad-b078-435b-9331-56bb8a431044'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:34.332783 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] The token cache is being invalidate because project e6b7311400ac4969825d2e5fa9898f76 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:40:34.333243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:34.333880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a9595dad-b078-435b-9331-56bb8a431044 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:34.335422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1173/2334] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:40:33 2022] DELETE /identity/v3/projects/e6b7311400ac4969825d2e5fa9898f76 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:34.341498 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc24021-74bb-45a7-a22a-368a875fa62b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.342099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc24021-74bb-45a7-a22a-368a875fa62b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.342099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc24021-74bb-45a7-a22a-368a875fa62b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.361893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bfc24021-74bb-45a7-a22a-368a875fa62b None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:34.363203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bfc24021-74bb-45a7-a22a-368a875fa62b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:34.409143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:34.424231 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1162/2335] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:34.433827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0c84e-5878-456f-95de-ce5fe11fbd50 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.434219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0c84e-5878-456f-95de-ce5fe11fbd50 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.434497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0c84e-5878-456f-95de-ce5fe11fbd50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.455773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-78e0c84e-5878-456f-95de-ce5fe11fbd50 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:34.456980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-78e0c84e-5878-456f-95de-ce5fe11fbd50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:34.465689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1174/2336] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:34.469534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc24021-74bb-45a7-a22a-368a875fa62b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:34.479401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc24021-74bb-45a7-a22a-368a875fa62b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:34.484240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:34.485161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.485611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.485897 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.487217 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:40:34.487530 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:34.489440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:34.505542 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1163/2337] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:40:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:40:34.506679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0c84e-5878-456f-95de-ce5fe11fbd50 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:34.516815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0c84e-5878-456f-95de-ce5fe11fbd50 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:34.522066 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:34.532464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.532464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.532464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.532464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:34.532464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:34.586212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1175/2338] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:40:34 2022] POST /identity/v3/projects => generated 385 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:40:34.590376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:34.600556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefea7be-9d3b-47b4-86a8-1773591a26ed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:34.663313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:34.663938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.664224 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.664840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.665163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:34.667160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:34.745242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1164/2339] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:40:34 2022] POST /identity/v3/users => generated 293 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:34.749678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:34.760707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b246c02a-a59d-49cf-8714-c5ec7215aeeb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:34.765246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:34.766045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.766426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.766711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.767359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:34.768959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:34.771256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:34.779596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1176/2340] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:40:34.783772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:34.794048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebdd824-1048-4c6a-a13d-bf34b0370343 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:34.799381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107776be-53aa-4c63-986f-058fd9f00575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:34.800020 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107776be-53aa-4c63-986f-058fd9f00575 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.800300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107776be-53aa-4c63-986f-058fd9f00575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.800597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107776be-53aa-4c63-986f-058fd9f00575 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.801255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107776be-53aa-4c63-986f-058fd9f00575 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:34.802954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107776be-53aa-4c63-986f-058fd9f00575 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:34.814667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1165/2341] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:40:34.818849 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:34.829175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268e67d-5317-4bac-9325-e85f26a9866b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:34.834019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:34.834675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.834950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.835230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] PATH_INFO: `/v3/projects/0c67969ba8d6431e8ef742b5761e09f0/users/bfce7649996c48f28fbce9f1b76254e0/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.851768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c67969ba8d6431e8ef742b5761e09f0, user_id=bfce7649996c48f28fbce9f1b76254e0, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:34.852824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:34.932646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1177/2342] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:40:34 2022] PUT /identity/v3/projects/0c67969ba8d6431e8ef742b5761e09f0/users/bfce7649996c48f28fbce9f1b76254e0/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:40:34.945911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7459dd3a-a911-4f62-a713-061857e4870e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:34.946203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7459dd3a-a911-4f62-a713-061857e4870e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:34.946490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7459dd3a-a911-4f62-a713-061857e4870e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:34.977783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7459dd3a-a911-4f62-a713-061857e4870e None None] MFA Rules not processed for user `bfce7649996c48f28fbce9f1b76254e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:34.979004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7459dd3a-a911-4f62-a713-061857e4870e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:35.048572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1166/2343] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:35.060355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:35.062232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:35.122116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9cef09-e41e-413a-b565-14c6db032002 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:35.125725 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:35.126318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:35.126602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:35.126878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:35.135945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:35.137407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:35.146551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1178/2344] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:35 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:41.082556 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9656f8ba-9e3f-4b19-becc-8dbd6f910700 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:41.083572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9656f8ba-9e3f-4b19-becc-8dbd6f910700 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:41.083841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9656f8ba-9e3f-4b19-becc-8dbd6f910700 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:41.103563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9656f8ba-9e3f-4b19-becc-8dbd6f910700 None None] MFA Rules not processed for user `fef6a70cbb564a55917ae1d9ae2f71e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:41.104759 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9656f8ba-9e3f-4b19-becc-8dbd6f910700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:41.147329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:41.172786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1167/2345] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:45.094406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:45.166211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81d09b9-5be0-4aaf-b67d-53978ba273b6 service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:45.170314 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:45.170981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:45.171236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:45.171515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] PATH_INFO: `/v3/users/bfce7649996c48f28fbce9f1b76254e0` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:45.172790 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] RBAC: Authorizing `identity:delete_user(user_id=bfce7649996c48f28fbce9f1b76254e0)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:45.174768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:45.406386 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bfce7649996c48f28fbce9f1b76254e0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:45.411673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bfce7649996c48f28fbce9f1b76254e0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:45.421574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bfce7649996c48f28fbce9f1b76254e0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:45.430260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bfce7649996c48f28fbce9f1b76254e0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:45.430985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bfce7649996c48f28fbce9f1b76254e0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:45.435483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:45.521591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1179/2346] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:40:45 2022] DELETE /identity/v3/users/bfce7649996c48f28fbce9f1b76254e0 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:45.526154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:45.526634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:45.526828 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:45.547595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:45.548810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:45.615267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1168/2347] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:40:45.625156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:45.626793 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:45.693194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0633366-b836-4d56-a914-b56500ab409b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:45.696834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6de838-7029-4d5a-87a7-0b17565c93a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:45.697511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6de838-7029-4d5a-87a7-0b17565c93a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:45.697797 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6de838-7029-4d5a-87a7-0b17565c93a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:45.698075 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6de838-7029-4d5a-87a7-0b17565c93a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:45.707315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6de838-7029-4d5a-87a7-0b17565c93a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:45.709546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6de838-7029-4d5a-87a7-0b17565c93a8 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:45.720752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1180/2348] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:40:46.303709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:46.305257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:46.318320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab77bad-7b46-471e-97a8-be779433d9c6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:46.323801 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:46.324394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.324785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.325118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] PATH_INFO: `/v3/projects/0c67969ba8d6431e8ef742b5761e09f0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.326305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c67969ba8d6431e8ef742b5761e09f0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:46.328274 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:46.431986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c67969ba8d6431e8ef742b5761e09f0', 'request_id': 'req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:46.509025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c67969ba8d6431e8ef742b5761e09f0', 'request_id': 'req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:46.516073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c67969ba8d6431e8ef742b5761e09f0', 'request_id': 'req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:46.526892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] The token cache is being invalidate because project 0c67969ba8d6431e8ef742b5761e09f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:40:46.529596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:46.529596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5fb176fe-35b2-4ef4-8f39-d6069f3c9468 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:40:46.529596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1169/2349] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:40:46 2022] DELETE /identity/v3/projects/0c67969ba8d6431e8ef742b5761e09f0 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:46.534481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914d7ff-9f9a-48c7-a6c0-89d41c196840 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.534955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914d7ff-9f9a-48c7-a6c0-89d41c196840 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.535057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914d7ff-9f9a-48c7-a6c0-89d41c196840 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.555262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-3914d7ff-9f9a-48c7-a6c0-89d41c196840 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:46.558083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3914d7ff-9f9a-48c7-a6c0-89d41c196840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:46.619147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1181/2350] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:46.626988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f71c74-4266-4375-a783-e4a0411f4904 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.627287 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f71c74-4266-4375-a783-e4a0411f4904 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.627573 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f71c74-4266-4375-a783-e4a0411f4904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.647014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-55f71c74-4266-4375-a783-e4a0411f4904 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:46.648216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-55f71c74-4266-4375-a783-e4a0411f4904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:46.656381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1170/2351] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:46.660669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914d7ff-9f9a-48c7-a6c0-89d41c196840 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:46.671393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914d7ff-9f9a-48c7-a6c0-89d41c196840 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:46.675978 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:46.677131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.677449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.677729 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.678377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:40:46.678678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:46.680277 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:46.691648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1182/2352] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:40:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:40:46.695865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f71c74-4266-4375-a783-e4a0411f4904 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:46.704958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f71c74-4266-4375-a783-e4a0411f4904 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:46.709951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:46.710553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.710835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.711110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.711741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:46.713674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:46.797031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1171/2353] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:40:46 2022] POST /identity/v3/projects => generated 369 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:46.800911 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:46.812011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a16510-0f6b-4f00-b0db-26252e0a458e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:46.870266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:46.870798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.871086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.871364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.872005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:46.874012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:46.927477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1183/2354] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:40:46 2022] POST /identity/v3/users => generated 285 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:40:46.931738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:46.943540 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66cde20-e814-4acb-9f73-7f5b48eacd86 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:46.952212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:46.952824 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.953110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.953424 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.954079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:46.955700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:46.967352 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1172/2355] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:40:46.972875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:46.984067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50872878-4024-4509-b891-b33d14c5fb2d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:46.988747 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbdefea-72a5-4f33-8dab-d01f3627f2dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:46.989831 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbdefea-72a5-4f33-8dab-d01f3627f2dd admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:46.990119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbdefea-72a5-4f33-8dab-d01f3627f2dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:46.990396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbdefea-72a5-4f33-8dab-d01f3627f2dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:46.991032 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbdefea-72a5-4f33-8dab-d01f3627f2dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:46.992650 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbdefea-72a5-4f33-8dab-d01f3627f2dd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:47.003302 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1184/2356] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:40:47.007501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:47.017620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d79013-f8c5-492c-bd36-d0c90bc3a621 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:47.022745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:47.023398 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:47.023683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:47.023964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] PATH_INFO: `/v3/projects/c6eb61c333054a1dabeef1576d550995/users/a57f15a4851640f0b14e4d0dcb662423/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:47.039245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6eb61c333054a1dabeef1576d550995, user_id=a57f15a4851640f0b14e4d0dcb662423, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:47.041938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:47.116129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1173/2357] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:40:47 2022] PUT /identity/v3/projects/c6eb61c333054a1dabeef1576d550995/users/a57f15a4851640f0b14e4d0dcb662423/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:40:47.126198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52f65c-3467-4f1b-a1a1-2ac1a5ba27fd None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:47.126485 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52f65c-3467-4f1b-a1a1-2ac1a5ba27fd None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:47.126950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb52f65c-3467-4f1b-a1a1-2ac1a5ba27fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:47.135674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:47.154763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-eb52f65c-3467-4f1b-a1a1-2ac1a5ba27fd None None] MFA Rules not processed for user `a57f15a4851640f0b14e4d0dcb662423`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:40:47.155957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-eb52f65c-3467-4f1b-a1a1-2ac1a5ba27fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:47.218891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1185/2358] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:40:47 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:40:47.230967 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:40:47.232664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:40:47.246205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:47.289640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a067dec-a711-452d-91f9-898b9b5087c1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:40:47.293172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7fae443ad92439aadd2f94ad3bca594', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:40:47.293805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:40:47.294088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:40:47.294376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:40:47.303135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:40:47.304549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:40:47.313222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1174/2359] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:40:47 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:40:57.225180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=30 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:40:57.320322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=21 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:06.684124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3957dc5-0437-4922-b95f-69fa609093c7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:06.684648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3957dc5-0437-4922-b95f-69fa609093c7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:06.684648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3957dc5-0437-4922-b95f-69fa609093c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:06.745172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-d3957dc5-0437-4922-b95f-69fa609093c7 None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:06.746481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-d3957dc5-0437-4922-b95f-69fa609093c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:06.813010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1186/2360] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:41:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:41:06.959483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:06.960902 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:07.054341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561cfa6-9186-4f91-8aaa-5adc07a779a6 service glance] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:07.059985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e573bd-c0ef-4386-a6b2-74549c769109 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:07.060597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e573bd-c0ef-4386-a6b2-74549c769109 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:07.060888 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e573bd-c0ef-4386-a6b2-74549c769109 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:07.061187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e573bd-c0ef-4386-a6b2-74549c769109 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:07.069491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e573bd-c0ef-4386-a6b2-74549c769109 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:07.071014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e573bd-c0ef-4386-a6b2-74549c769109 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:07.078977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1175/2361] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:41:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:41:16.810161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=44 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:17.075856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=41 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:21.463402 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3957dc5-0437-4922-b95f-69fa609093c7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:21.579008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3957dc5-0437-4922-b95f-69fa609093c7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:21.587663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:21.587663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:21.587663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:21.587663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] PATH_INFO: `/v3/users/a57f15a4851640f0b14e4d0dcb662423` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:21.588057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=a57f15a4851640f0b14e4d0dcb662423)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:21.588712 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:21.828804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a57f15a4851640f0b14e4d0dcb662423'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:21.829656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a57f15a4851640f0b14e4d0dcb662423'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:21.845131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a57f15a4851640f0b14e4d0dcb662423'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:21.853635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a57f15a4851640f0b14e4d0dcb662423'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:21.854391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a57f15a4851640f0b14e4d0dcb662423'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:21.974583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1187/2362] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:41:21 2022] DELETE /identity/v3/users/a57f15a4851640f0b14e4d0dcb662423 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:41:21.979497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:21.980041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:21.980041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:22.033479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:22.034728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:22.123903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1176/2363] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:22.130678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:22.133102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:22.209357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e361df7-bfa4-41ed-ae20-95b4639fd5ef admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:22.216590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9341eecd-4238-4c36-9b4a-6f2d870c9cf7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:22.217223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9341eecd-4238-4c36-9b4a-6f2d870c9cf7 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:22.217562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9341eecd-4238-4c36-9b4a-6f2d870c9cf7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:22.217821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9341eecd-4238-4c36-9b4a-6f2d870c9cf7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:22.232326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9341eecd-4238-4c36-9b4a-6f2d870c9cf7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:22.233845 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9341eecd-4238-4c36-9b4a-6f2d870c9cf7 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:22.246728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1188/2364] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:41:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:41:22.757305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:22.758851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:22.770870 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc2fe09-9aaa-4023-b352-526da818fbba None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:22.777798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:22.778476 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:22.778806 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:22.779027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] PATH_INFO: `/v3/projects/c6eb61c333054a1dabeef1576d550995` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:22.780215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6eb61c333054a1dabeef1576d550995)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:22.783599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:22.911012 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6eb61c333054a1dabeef1576d550995', 'request_id': 'req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:22.977979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6eb61c333054a1dabeef1576d550995', 'request_id': 'req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:22.984560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6eb61c333054a1dabeef1576d550995', 'request_id': 'req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:22.995408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] The token cache is being invalidate because project c6eb61c333054a1dabeef1576d550995 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:41:22.995699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:22.996346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3f259e9b-df2d-49f4-a15d-0e8fd2c12c2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:22.997859 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1177/2365] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:41:22 2022] DELETE /identity/v3/projects/c6eb61c333054a1dabeef1576d550995 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:41:23.002835 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7c21a9-5681-4e80-8b1b-909aec80f0a7 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.005871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7c21a9-5681-4e80-8b1b-909aec80f0a7 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.005871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7c21a9-5681-4e80-8b1b-909aec80f0a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.009406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:23.053823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6e7c21a9-5681-4e80-8b1b-909aec80f0a7 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:23.055409 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6e7c21a9-5681-4e80-8b1b-909aec80f0a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:23.116656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1189/2366] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:23.124527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fade76-dd16-42d6-915a-66553c189dbb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.124840 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fade76-dd16-42d6-915a-66553c189dbb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.125138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fade76-dd16-42d6-915a-66553c189dbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.145411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f7fade76-dd16-42d6-915a-66553c189dbb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:23.146723 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f7fade76-dd16-42d6-915a-66553c189dbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:23.156242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1178/2367] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:23.160366 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7c21a9-5681-4e80-8b1b-909aec80f0a7 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.170817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7c21a9-5681-4e80-8b1b-909aec80f0a7 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.175472 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.176067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.176347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.177027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.177740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:41:23.178109 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.179860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.191309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1190/2368] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:41:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:41:23.195620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fade76-dd16-42d6-915a-66553c189dbb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.204961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fade76-dd16-42d6-915a-66553c189dbb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.210375 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16808ef4-e7cf-41c7-8fc1-70f5a2498d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.211201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16808ef4-e7cf-41c7-8fc1-70f5a2498d18 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.211492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16808ef4-e7cf-41c7-8fc1-70f5a2498d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.211770 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16808ef4-e7cf-41c7-8fc1-70f5a2498d18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.212405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16808ef4-e7cf-41c7-8fc1-70f5a2498d18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.214471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16808ef4-e7cf-41c7-8fc1-70f5a2498d18 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.308810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1179/2369] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:41:23 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:41:23.310591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.321848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7b0699-5633-480d-937d-b131c40cefd9 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.380037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.380679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.380974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.381226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.382416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.384511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.431562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224d2d36-738d-4b84-ba88-c754eb602424 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.431923 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224d2d36-738d-4b84-ba88-c754eb602424 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.432206 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224d2d36-738d-4b84-ba88-c754eb602424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.452421 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-224d2d36-738d-4b84-ba88-c754eb602424 None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:23.453656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-224d2d36-738d-4b84-ba88-c754eb602424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:23.479167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:23.485243 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1191/2370] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:41:23 2022] POST /identity/v3/users => generated 287 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:41:23.486678 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.497544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755cc5ef-387c-4755-8879-aba3d331d4f0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.502493 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.503186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.503532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.503886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.504597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.506299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.527002 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1180/2372] 10.222.0.66 () {58 vars in 967 bytes} [Fri Oct 28 20:41:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:41:23.528335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1192/2371] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:41:23.530422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224d2d36-738d-4b84-ba88-c754eb602424 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.545022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.545022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:23.545215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224d2d36-738d-4b84-ba88-c754eb602424 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.548412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.549075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.550319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.551013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.552054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.553779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.566157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1181/2373] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:41:23.569673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.579585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304f30e-60bc-4fe1-a387-943757c26e6a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.584534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.586172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.586455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.586736 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] PATH_INFO: `/v3/projects/6d8416b8a3a7493b941261eb5297a798/users/9835cd7baa6b412398797dcdd63db416/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.592984 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:23.603351 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d8416b8a3a7493b941261eb5297a798, user_id=9835cd7baa6b412398797dcdd63db416, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.606067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.614415 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aee503-fe4c-4e94-9281-5ac7fe818eb0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.621028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5e2450-0e52-4bb5-9af5-690550214855 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.621771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5e2450-0e52-4bb5-9af5-690550214855 service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.622083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5e2450-0e52-4bb5-9af5-690550214855 service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.622363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5e2450-0e52-4bb5-9af5-690550214855 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.630610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5e2450-0e52-4bb5-9af5-690550214855 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.632257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5e2450-0e52-4bb5-9af5-690550214855 service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.640551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1193/2374] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:41:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 20:41:23.671214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1182/2375] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:41:23 2022] PUT /identity/v3/projects/6d8416b8a3a7493b941261eb5297a798/users/9835cd7baa6b412398797dcdd63db416/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:41:23.682847 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4390a0-b2ea-4c24-b3c4-ed8a3f38428a None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.683138 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4390a0-b2ea-4c24-b3c4-ed8a3f38428a None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.683420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4390a0-b2ea-4c24-b3c4-ed8a3f38428a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.712034 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-bf4390a0-b2ea-4c24-b3c4-ed8a3f38428a None None] MFA Rules not processed for user `9835cd7baa6b412398797dcdd63db416`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:23.713255 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-bf4390a0-b2ea-4c24-b3c4-ed8a3f38428a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:23.778886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1194/2376] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:41:23.790497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:23.792154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:23.850497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22cbaf5-a641-4c26-b31b-e7d038b51e28 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:23.854120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:23.854734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:23.855017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:23.855291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:23.863690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:23.865482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:23.875188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1183/2377] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:41:23 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:41:25.429026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff118179-0724-4379-9fc7-cc77ba30d4d6 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:25.429563 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff118179-0724-4379-9fc7-cc77ba30d4d6 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:25.429668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff118179-0724-4379-9fc7-cc77ba30d4d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:25.450010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ff118179-0724-4379-9fc7-cc77ba30d4d6 None None] MFA Rules not processed for user `9835cd7baa6b412398797dcdd63db416`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:25.451198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ff118179-0724-4379-9fc7-cc77ba30d4d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:25.458020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1195/2378] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:26.505273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:26.506903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:26.578288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4608c9-3ba3-4446-9275-1e97df2252e4 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:26.582827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:26.583461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:26.583741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:26.584013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] PATH_INFO: `/v3/users/fef6a70cbb564a55917ae1d9ae2f71e1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:26.585130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fef6a70cbb564a55917ae1d9ae2f71e1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:26.588179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:26.773065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:26.776200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fef6a70cbb564a55917ae1d9ae2f71e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:26.788298 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fef6a70cbb564a55917ae1d9ae2f71e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:26.847480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fef6a70cbb564a55917ae1d9ae2f71e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:26.848168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fef6a70cbb564a55917ae1d9ae2f71e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:26.848782 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ee0f726c-ca8b-42d4-a9b8-9ffb1a96d9f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fef6a70cbb564a55917ae1d9ae2f71e1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:26.861947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1184/2379] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:41:26 2022] DELETE /identity/v3/users/fef6a70cbb564a55917ae1d9ae2f71e1 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:41:27.106244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff118179-0724-4379-9fc7-cc77ba30d4d6 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:27.123864 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff118179-0724-4379-9fc7-cc77ba30d4d6 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:27.187804 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:27.188439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.188721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.189036 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] PATH_INFO: `/v3/projects/40b9acd32a1f4bbcb3bf71ac93bae1f5` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.190267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40b9acd32a1f4bbcb3bf71ac93bae1f5)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:27.192235 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:27.288256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40b9acd32a1f4bbcb3bf71ac93bae1f5', 'request_id': 'req-6e43b068-5e5c-4128-bf69-c98d307938f9'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:27.294363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40b9acd32a1f4bbcb3bf71ac93bae1f5', 'request_id': 'req-6e43b068-5e5c-4128-bf69-c98d307938f9'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:27.296202 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:27.300851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40b9acd32a1f4bbcb3bf71ac93bae1f5', 'request_id': 'req-6e43b068-5e5c-4128-bf69-c98d307938f9'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:27.349604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] The token cache is being invalidate because project 40b9acd32a1f4bbcb3bf71ac93bae1f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:41:27.349898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:27.350533 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6e43b068-5e5c-4128-bf69-c98d307938f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:27.352043 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1196/2380] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:41:27 2022] DELETE /identity/v3/projects/40b9acd32a1f4bbcb3bf71ac93bae1f5 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:41:27.357646 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cbeb7a-d2bc-4955-8563-9663c784a3da None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.357938 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cbeb7a-d2bc-4955-8563-9663c784a3da None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.358220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cbeb7a-d2bc-4955-8563-9663c784a3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.378330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-72cbeb7a-d2bc-4955-8563-9663c784a3da None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:27.379553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-72cbeb7a-d2bc-4955-8563-9663c784a3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:27.445203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1185/2381] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:27.452753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995b5eca-f828-4b72-ba88-f83de5bd0268 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.453044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995b5eca-f828-4b72-ba88-f83de5bd0268 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.453331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995b5eca-f828-4b72-ba88-f83de5bd0268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.473470 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-995b5eca-f828-4b72-ba88-f83de5bd0268 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:27.474752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-995b5eca-f828-4b72-ba88-f83de5bd0268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:27.483363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1197/2382] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:27.487832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cbeb7a-d2bc-4955-8563-9663c784a3da None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:27.498657 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cbeb7a-d2bc-4955-8563-9663c784a3da None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:27.503378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:27.503993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.504280 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.504560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.505219 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:41:27.505574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:27.507439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:27.525059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1186/2383] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:41:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:41:27.526347 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995b5eca-f828-4b72-ba88-f83de5bd0268 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:27.535696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995b5eca-f828-4b72-ba88-f83de5bd0268 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:27.541266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:27.541836 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.542130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.542410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.543040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:27.544965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:27.613719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1198/2384] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:41:27 2022] POST /identity/v3/projects => generated 397 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:41:27.618130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:27.628222 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94361b9-8aef-4d22-b839-4d671626d3f2 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:27.684928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:27.685569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.685853 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.686129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.686751 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:27.688711 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:27.750211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1187/2385] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:41:27 2022] POST /identity/v3/users => generated 299 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:41:27.754476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:27.765779 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09059845-511e-4acf-a466-409c3e7c9446 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:27.770435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:27.771037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.771325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.771599 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.772239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:27.773879 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:27.785372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1199/2386] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:41:27.789099 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:27.799596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c20832-13db-465e-b784-5748c198c828 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:27.805735 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c9db6c-1da9-438f-bfa8-1a89c49e0ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:27.806528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c9db6c-1da9-438f-bfa8-1a89c49e0ed4 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.806809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c9db6c-1da9-438f-bfa8-1a89c49e0ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.807082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c9db6c-1da9-438f-bfa8-1a89c49e0ed4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.807783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c9db6c-1da9-438f-bfa8-1a89c49e0ed4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:27.809393 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c9db6c-1da9-438f-bfa8-1a89c49e0ed4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:27.822270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1188/2387] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:41:27.826424 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:27.836223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d069d94-bff7-458e-a739-96e112f0cc43 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:27.841808 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:27.842453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.842734 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.843009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] PATH_INFO: `/v3/projects/9423a4223c2e41dba994330efdf885e5/users/565e46a8776345549566f3f8456d1baa/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.857992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] RBAC: Authorizing `identity:create_grant(project_id=9423a4223c2e41dba994330efdf885e5, user_id=565e46a8776345549566f3f8456d1baa, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:27.860679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:27.944265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1200/2388] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:41:27 2022] PUT /identity/v3/projects/9423a4223c2e41dba994330efdf885e5/users/565e46a8776345549566f3f8456d1baa/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:41:27.955394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0f8be-c49d-4d66-b730-dfd60471bea5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:27.955687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0f8be-c49d-4d66-b730-dfd60471bea5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:27.955975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0f8be-c49d-4d66-b730-dfd60471bea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:27.988271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-08e0f8be-c49d-4d66-b730-dfd60471bea5 None None] MFA Rules not processed for user `565e46a8776345549566f3f8456d1baa`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:27.989647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-08e0f8be-c49d-4d66-b730-dfd60471bea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:28.003656 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:28.061346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1189/2389] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:27 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:28.075198 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:28.077028 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:28.140741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cea9206-f524-4ffa-bbdb-631117aa94be admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:28.145207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be4105-67f9-4dd9-8e3b-f18108a10411 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:28.145688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be4105-67f9-4dd9-8e3b-f18108a10411 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:28.145973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be4105-67f9-4dd9-8e3b-f18108a10411 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:28.146253 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be4105-67f9-4dd9-8e3b-f18108a10411 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:28.158248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be4105-67f9-4dd9-8e3b-f18108a10411 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:28.159698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be4105-67f9-4dd9-8e3b-f18108a10411 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:28.169428 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1201/2390] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:41:28 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:41:37.143049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e0f8be-c49d-4d66-b730-dfd60471bea5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:37.205125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e0f8be-c49d-4d66-b730-dfd60471bea5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:37.209783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:37.210427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:37.210716 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:37.210993 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] PATH_INFO: `/v3/users/565e46a8776345549566f3f8456d1baa` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:37.212115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=565e46a8776345549566f3f8456d1baa)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:37.214325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:37.370319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '565e46a8776345549566f3f8456d1baa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:37.382209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '565e46a8776345549566f3f8456d1baa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:37.435007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '565e46a8776345549566f3f8456d1baa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:37.435639 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '565e46a8776345549566f3f8456d1baa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:37.436262 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '565e46a8776345549566f3f8456d1baa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:37.449380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1190/2391] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:41:37 2022] DELETE /identity/v3/users/565e46a8776345549566f3f8456d1baa => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:41:37.454837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:37.455349 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:37.455745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:37.477224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:37.478440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:37.537086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1202/2392] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:37.547454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:37.548843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:37.588927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:37.606482 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa1b8d2-2e5f-4864-8f4d-8fb3dc219e2f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:37.610195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e01213-d420-4d70-a2eb-183d2d253e26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:37.612700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e01213-d420-4d70-a2eb-183d2d253e26 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:37.612700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e01213-d420-4d70-a2eb-183d2d253e26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:37.612700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e01213-d420-4d70-a2eb-183d2d253e26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:37.621473 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e01213-d420-4d70-a2eb-183d2d253e26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:37.622919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e01213-d420-4d70-a2eb-183d2d253e26 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:37.632707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1191/2393] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:41:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:41:38.013991 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:38.015592 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:38.032179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b917ba7-0e10-4568-a333-e0ccb3ffeecb None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:38.038323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:38.048118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:38.048118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:38.048118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] PATH_INFO: `/v3/projects/9423a4223c2e41dba994330efdf885e5` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:38.048118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=9423a4223c2e41dba994330efdf885e5)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:38.048118 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:38.073487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:38.144663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9423a4223c2e41dba994330efdf885e5', 'request_id': 'req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:38.151602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9423a4223c2e41dba994330efdf885e5', 'request_id': 'req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:38.158716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9423a4223c2e41dba994330efdf885e5', 'request_id': 'req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:38.229700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] The token cache is being invalidate because project 9423a4223c2e41dba994330efdf885e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:41:38.230039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:38.230675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:38.232217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1203/2394] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:41:38 2022] DELETE /identity/v3/projects/9423a4223c2e41dba994330efdf885e5 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:41:45.475301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c0e82-3ede-438a-8a9b-72d928cf2624 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:45.475789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c0e82-3ede-438a-8a9b-72d928cf2624 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:45.475885 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c0e82-3ede-438a-8a9b-72d928cf2624 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:45.496200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-230c0e82-3ede-438a-8a9b-72d928cf2624 None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:45.497757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-230c0e82-3ede-438a-8a9b-72d928cf2624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:45.553137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1192/2395] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:41:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:45.565839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:45.567216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:45.636315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05d66f-65cf-4dc1-a6b8-04020d4956cb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:45.641770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:45.642372 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:45.648229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:45.648229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:45.650960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:45.652575 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:45.660483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1204/2396] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:41:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 20:41:46.498040 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cbc39a-2bc0-4cfc-940f-e69adb43ed3c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:46.498500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cbc39a-2bc0-4cfc-940f-e69adb43ed3c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:46.498601 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cbc39a-2bc0-4cfc-940f-e69adb43ed3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:46.519304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-01cbc39a-2bc0-4cfc-940f-e69adb43ed3c None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:46.520708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-01cbc39a-2bc0-4cfc-940f-e69adb43ed3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:46.528606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1193/2397] 10.222.0.66 () {58 vars in 967 bytes} [Fri Oct 28 20:41:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:46.540583 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:46.551231 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1978608b-e7d0-416b-aa19-7c8a9de1f568 service swift] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:46.556183 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:46.556770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:46.557050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:46.557341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:46.566046 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:46.567639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:46.576508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1205/2398] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:41:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:41:47.144191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425caae-8622-4c55-b7cb-5490cef049de None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:47.144483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425caae-8622-4c55-b7cb-5490cef049de None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:47.144765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425caae-8622-4c55-b7cb-5490cef049de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:47.163942 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d425caae-8622-4c55-b7cb-5490cef049de None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:47.165134 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d425caae-8622-4c55-b7cb-5490cef049de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:47.173403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1194/2399] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:41:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:47.185718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:47.195936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc351d83-dcf4-48a9-8b2b-d9d105bb87c4 service swift] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:47.201042 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:47.201651 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:47.201931 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:47.202200 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:47.209951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:47.211508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:47.219458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1206/2400] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:41:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:41:47.797259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:47.797857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:47.797857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:47.816700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:47.817926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:47.825235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1195/2401] 10.222.0.66 () {58 vars in 967 bytes} [Fri Oct 28 20:41:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:47.838021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:47.849122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07f6fee-2ca9-4ae4-abaf-6a15e3d3f4dd service swift] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:47.854217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c843604f-ce1d-4097-88dc-ef4352c10b2e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:47.854813 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c843604f-ce1d-4097-88dc-ef4352c10b2e service swift] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:47.855680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c843604f-ce1d-4097-88dc-ef4352c10b2e service swift] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:47.855964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c843604f-ce1d-4097-88dc-ef4352c10b2e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:47.863878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c843604f-ce1d-4097-88dc-ef4352c10b2e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:47.865494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c843604f-ce1d-4097-88dc-ef4352c10b2e service swift] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:47.874055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1207/2402] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:41:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:41:50.329722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:50.331114 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:50.394811 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb14e72b-aa75-4160-ac23-2d55bdd0e4c5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:50.399294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:50.399914 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:50.400197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:50.400472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] PATH_INFO: `/v3/users/9835cd7baa6b412398797dcdd63db416` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:50.401596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=9835cd7baa6b412398797dcdd63db416)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:50.403524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:50.586328 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9835cd7baa6b412398797dcdd63db416'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:50.598827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9835cd7baa6b412398797dcdd63db416'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:50.600497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:50.652886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9835cd7baa6b412398797dcdd63db416'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:50.653561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9835cd7baa6b412398797dcdd63db416'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:50.654175 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9835cd7baa6b412398797dcdd63db416'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:50.666664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1196/2403] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:41:50 2022] DELETE /identity/v3/users/9835cd7baa6b412398797dcdd63db416 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:41:50.671874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:50.672361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:50.672466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:50.692461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:41:50.693761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:50.754973 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1208/2404] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:41:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:41:50.764715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:50.766187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:50.823093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9741fa1-7bed-44bf-8a09-0a62114390bb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:50.827420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:50.828031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:50.828319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:50.828600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:50.837634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:50.838996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:50.849490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1197/2405] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:41:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:41:51.095176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:41:51.096865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:41:51.119210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:41:51.119210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da95fef3-cf83-45ae-90cf-9cca469ec5d2 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:41:51.119210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:41:51.119210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:41:51.120378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:41:51.120378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] PATH_INFO: `/v3/projects/6d8416b8a3a7493b941261eb5297a798` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:41:51.120378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d8416b8a3a7493b941261eb5297a798)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:41:51.120378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:41:51.236097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d8416b8a3a7493b941261eb5297a798', 'request_id': 'req-39b2217a-f813-481d-bc19-ba3daef2cd94'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:51.242008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d8416b8a3a7493b941261eb5297a798', 'request_id': 'req-39b2217a-f813-481d-bc19-ba3daef2cd94'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:51.248908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d8416b8a3a7493b941261eb5297a798', 'request_id': 'req-39b2217a-f813-481d-bc19-ba3daef2cd94'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:51.303251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] The token cache is being invalidate because project 6d8416b8a3a7493b941261eb5297a798 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:41:51.303541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:51.304159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:41:51.305681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1209/2406] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:41:51 2022] DELETE /identity/v3/projects/6d8416b8a3a7493b941261eb5297a798 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:42:00.180584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:42:00.182009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:42:00.239729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facddafd-5d65-4246-8088-36dc4af3821a service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:42:00.289213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:42:00.289855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:42:00.290137 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:42:00.290410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:42:00.292112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:42:00.351317 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=98 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:42:00.365511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:42:00.367214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:42:00.376892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1198/2407] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:42:00 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:42:01.300089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=26,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:42:03.786519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:42:03.788073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:42:03.801798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b2217a-f813-481d-bc19-ba3daef2cd94 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:42:03.805433 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:42:03.806029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:42:03.806316 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:42:03.806589 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:42:03.808249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:42:03.888477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:42:03.889959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:42:03.908247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1210/2408] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:42:03 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:42:10.436068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:42:13.905755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=41 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:05.840758 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:05.842201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:05.899763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d103c56-2552-4119-bc61-5471a663ff36 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:05.903353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:05.903959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:05.904244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:05.904523 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:05.906237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:05.967329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:05.968691 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:05.978289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1199/2409] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:09.544646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:09.546901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:09.565513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8477af7-ef97-44d7-9060-e0213f5a4f4c service cinder] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:09.570306 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:09.571123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:09.571376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:09.571658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] PATH_INFO: `/v3/users/15db5106efbd434f99a4d102810cc7a6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:09.572878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=15db5106efbd434f99a4d102810cc7a6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:09.574867 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:09.798663 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15db5106efbd434f99a4d102810cc7a6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:09.799487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15db5106efbd434f99a4d102810cc7a6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:09.811669 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15db5106efbd434f99a4d102810cc7a6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:09.819404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15db5106efbd434f99a4d102810cc7a6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:09.820040 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7bb3ff06-5891-4dbc-a114-cc5243687b29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15db5106efbd434f99a4d102810cc7a6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:09.886274 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1211/2410] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:09 2022] DELETE /identity/v3/users/15db5106efbd434f99a4d102810cc7a6 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:10.129490 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:10.139290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984fc54-d65f-428a-8396-d0d9ab3cfec5 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:10.202927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:10.205074 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:10.205395 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:10.215953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] PATH_INFO: `/v3/projects/8b163bdfc7544a3d833bd69dc027c7d6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:10.215953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b163bdfc7544a3d833bd69dc027c7d6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:10.215953 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:10.324205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b163bdfc7544a3d833bd69dc027c7d6', 'request_id': 'req-ac335b73-26ad-4ed1-aee1-9341c0547aea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:10.326737 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:10.424135 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b163bdfc7544a3d833bd69dc027c7d6', 'request_id': 'req-ac335b73-26ad-4ed1-aee1-9341c0547aea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:10.431836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b163bdfc7544a3d833bd69dc027c7d6', 'request_id': 'req-ac335b73-26ad-4ed1-aee1-9341c0547aea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:10.447730 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] The token cache is being invalidate because project 8b163bdfc7544a3d833bd69dc027c7d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:10.448029 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:10.448727 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac335b73-26ad-4ed1-aee1-9341c0547aea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:10.450422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1200/2411] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:10 2022] DELETE /identity/v3/projects/8b163bdfc7544a3d833bd69dc027c7d6 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:10.457602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb21d48-0a7b-4f65-a9cc-193d0117e324 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:10.457909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb21d48-0a7b-4f65-a9cc-193d0117e324 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:10.458193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb21d48-0a7b-4f65-a9cc-193d0117e324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:10.483715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2cb21d48-0a7b-4f65-a9cc-193d0117e324 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:10.484935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2cb21d48-0a7b-4f65-a9cc-193d0117e324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:10.558499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1212/2412] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:10.570387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ef62b-ba57-4d5f-bf14-2a09868c61eb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:10.570677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ef62b-ba57-4d5f-bf14-2a09868c61eb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:10.571014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ef62b-ba57-4d5f-bf14-2a09868c61eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:10.592479 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7e6ef62b-ba57-4d5f-bf14-2a09868c61eb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:10.593772 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7e6ef62b-ba57-4d5f-bf14-2a09868c61eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:10.604275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1201/2413] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:10.609077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb21d48-0a7b-4f65-a9cc-193d0117e324 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:10.621865 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb21d48-0a7b-4f65-a9cc-193d0117e324 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:10.627551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:10.628369 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:10.628648 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:10.628924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:10.629604 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:43:10.629908 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:10.631687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:10.648350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1213/2414] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:43:10.649899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ef62b-ba57-4d5f-bf14-2a09868c61eb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:10.663951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ef62b-ba57-4d5f-bf14-2a09868c61eb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:10.670715 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:10.671379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:10.671662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:10.673216 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:10.673907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:10.676191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:10.775581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1202/2415] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:10 2022] POST /identity/v3/projects => generated 403 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:10.780195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:10.793591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034116c1-2d12-4d19-88b9-7123fa114245 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:10.867250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:10.867858 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:10.868406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:10.868703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:10.869612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:10.873511 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:10.882756 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:10.978780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1214/2416] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:10 2022] POST /identity/v3/users => generated 302 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:10.982952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:10.995909 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac48e944-0ed6-4a23-9d3d-500e1784264a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.000719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.002004 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.002367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.002671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.003320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.004957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.016864 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1203/2417] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:43:11.021010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.031166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc09fb1-9188-42b9-9499-894214193d26 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.036363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9225e4eb-81b2-41b5-b135-5db5ba079609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.036964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9225e4eb-81b2-41b5-b135-5db5ba079609 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.037248 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9225e4eb-81b2-41b5-b135-5db5ba079609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.037555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9225e4eb-81b2-41b5-b135-5db5ba079609 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.038197 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9225e4eb-81b2-41b5-b135-5db5ba079609 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.039791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9225e4eb-81b2-41b5-b135-5db5ba079609 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.051055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1215/2418] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:11.055242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.066439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2460da5f-7390-43f7-ba39-3dbb0275fe80 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.071006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.071660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.071940 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.072214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] PATH_INFO: `/v3/projects/c09258f37cc6475a83a2b1feabc1bcf3/users/3633b153922d4406be40eebdb5e2e865/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.086907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c09258f37cc6475a83a2b1feabc1bcf3, user_id=3633b153922d4406be40eebdb5e2e865, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.089744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.149717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1204/2419] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:11 2022] PUT /identity/v3/projects/c09258f37cc6475a83a2b1feabc1bcf3/users/3633b153922d4406be40eebdb5e2e865/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:11.165787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272045e-db15-4c02-ab54-054ec3622e42 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.166072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272045e-db15-4c02-ab54-054ec3622e42 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.166346 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272045e-db15-4c02-ab54-054ec3622e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.194933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5272045e-db15-4c02-ab54-054ec3622e42 None None] MFA Rules not processed for user `3633b153922d4406be40eebdb5e2e865`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:11.196132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5272045e-db15-4c02-ab54-054ec3622e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:11.260092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1216/2420] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:11 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:11.264616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.326471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a2d2e1-9b62-4cc4-8ed5-9ba83aefc6d6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.331635 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.332246 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.332532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.332810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.333493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.335428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.430611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1205/2421] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:11 2022] POST /identity/v3/projects => generated 403 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:11.434595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5272045e-db15-4c02-ab54-054ec3622e42 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.445846 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5272045e-db15-4c02-ab54-054ec3622e42 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.502647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.503261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.503544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.503822 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.504458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.506421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.589281 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1217/2422] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:11 2022] POST /identity/v3/users => generated 308 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:11.593547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.604581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051adfa6-3cca-40e5-9d46-5e0d3dae9f86 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.609338 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.609957 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.611034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.611539 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.612309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.613945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.624826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1206/2423] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:11.629351 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.639379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0338d2e-1713-418b-ac65-0cdbd1172f50 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.644895 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.645584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.645871 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.646151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] PATH_INFO: `/v3/projects/a4a120e493c4480093217fbdadf469e9/users/7d6e6fde7c494dfea3c9a7e5b29e5e1d/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.661850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4a120e493c4480093217fbdadf469e9, user_id=7d6e6fde7c494dfea3c9a7e5b29e5e1d, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.664525 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.738916 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1218/2424] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:11 2022] PUT /identity/v3/projects/a4a120e493c4480093217fbdadf469e9/users/7d6e6fde7c494dfea3c9a7e5b29e5e1d/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:11.742768 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.753422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863b047-a893-4031-a885-d20594e61d88 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.806901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:11.816929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4185c3a3-7479-49ab-972e-6dde69a6912e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.820372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185c3a3-7479-49ab-972e-6dde69a6912e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.820372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185c3a3-7479-49ab-972e-6dde69a6912e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.820372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185c3a3-7479-49ab-972e-6dde69a6912e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.820372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4185c3a3-7479-49ab-972e-6dde69a6912e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.820372 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4185c3a3-7479-49ab-972e-6dde69a6912e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.830387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1207/2425] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:11 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:11.834843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.845705 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb9d5dd-bfeb-4b53-bf65-ebd3b7ac12d5 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:11.850464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:11.851124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.851407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.851681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] PATH_INFO: `/v3/projects/a4a120e493c4480093217fbdadf469e9/users/7d6e6fde7c494dfea3c9a7e5b29e5e1d/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.853679 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4a120e493c4480093217fbdadf469e9, user_id=7d6e6fde7c494dfea3c9a7e5b29e5e1d, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:11.856403 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:11.866748 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1219/2426] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:11 2022] PUT /identity/v3/projects/a4a120e493c4480093217fbdadf469e9/users/7d6e6fde7c494dfea3c9a7e5b29e5e1d/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:11.876790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d79626c-21b1-4db1-b1de-325a0bdd4acb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:11.877081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d79626c-21b1-4db1-b1de-325a0bdd4acb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:11.877386 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d79626c-21b1-4db1-b1de-325a0bdd4acb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:11.906617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3d79626c-21b1-4db1-b1de-325a0bdd4acb None None] MFA Rules not processed for user `7d6e6fde7c494dfea3c9a7e5b29e5e1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:11.908438 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3d79626c-21b1-4db1-b1de-325a0bdd4acb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:11.974423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1208/2427] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:11 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:11.984752 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:11.986519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:12.009844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:12.056896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1f51d0-eae2-45f2-a95d-3399420fa2ed admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:12.060491 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41f7f3-676b-47ac-8494-efc357b26836 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:12.061497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41f7f3-676b-47ac-8494-efc357b26836 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:12.061497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41f7f3-676b-47ac-8494-efc357b26836 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:12.061681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41f7f3-676b-47ac-8494-efc357b26836 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:12.070826 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c41f7f3-676b-47ac-8494-efc357b26836 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:12.072244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c41f7f3-676b-47ac-8494-efc357b26836 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:12.082551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1220/2428] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:11 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:12.215686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d79626c-21b1-4db1-b1de-325a0bdd4acb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:12.279040 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d79626c-21b1-4db1-b1de-325a0bdd4acb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:12.284180 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:12.284814 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:12.285105 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:12.285418 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] PATH_INFO: `/v3/users/3633b153922d4406be40eebdb5e2e865` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:12.286541 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] RBAC: Authorizing `identity:delete_user(user_id=3633b153922d4406be40eebdb5e2e865)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:12.288456 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:12.512917 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3633b153922d4406be40eebdb5e2e865'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:12.524958 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3633b153922d4406be40eebdb5e2e865'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:12.606536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3633b153922d4406be40eebdb5e2e865'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:12.607096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3633b153922d4406be40eebdb5e2e865'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:12.607704 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3633b153922d4406be40eebdb5e2e865'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:12.610741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:12.621933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1209/2429] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:12 2022] DELETE /identity/v3/users/3633b153922d4406be40eebdb5e2e865 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:12.627050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:12.627535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:12.627637 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:12.646404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:12.653733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:12.707772 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1221/2430] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:12.717651 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:12.719104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:12.780366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010103ed-928b-4b37-8a7c-dfc5e4973bdc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:12.784052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:12.784665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:12.784951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:12.785228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:12.794213 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:12.795676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:12.806247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1210/2431] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:13.185063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:13.186620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:13.201297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e63457-8f0c-4a87-9929-3608deed0fb6 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:13.206848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:13.207497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:13.207786 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:13.208066 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] PATH_INFO: `/v3/projects/c09258f37cc6475a83a2b1feabc1bcf3` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:13.209199 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c09258f37cc6475a83a2b1feabc1bcf3)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:13.211185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:13.332753 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c09258f37cc6475a83a2b1feabc1bcf3', 'request_id': 'req-70a651d7-a1c8-4097-a9c6-5690f494f485'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.338330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c09258f37cc6475a83a2b1feabc1bcf3', 'request_id': 'req-70a651d7-a1c8-4097-a9c6-5690f494f485'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.344574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c09258f37cc6475a83a2b1feabc1bcf3', 'request_id': 'req-70a651d7-a1c8-4097-a9c6-5690f494f485'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.417130 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] The token cache is being invalidate because project c09258f37cc6475a83a2b1feabc1bcf3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:13.417447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.418072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.419558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1222/2432] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:13 2022] DELETE /identity/v3/projects/c09258f37cc6475a83a2b1feabc1bcf3 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:13.424404 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:13.434239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c806a96-c0ae-4e57-aa44-6c145ba0290b service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:13.491729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:13.492376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:13.492655 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:13.493197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] PATH_INFO: `/v3/users/7d6e6fde7c494dfea3c9a7e5b29e5e1d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:13.494423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d6e6fde7c494dfea3c9a7e5b29e5e1d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:13.496401 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:13.666229 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d6e6fde7c494dfea3c9a7e5b29e5e1d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.679117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d6e6fde7c494dfea3c9a7e5b29e5e1d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.754789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d6e6fde7c494dfea3c9a7e5b29e5e1d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.755462 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d6e6fde7c494dfea3c9a7e5b29e5e1d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.756079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-cb3070e5-5f0c-4089-8a42-7ddd433acf91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d6e6fde7c494dfea3c9a7e5b29e5e1d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:13.769363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1211/2433] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:13 2022] DELETE /identity/v3/users/7d6e6fde7c494dfea3c9a7e5b29e5e1d => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:14.171862 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:14.174333 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:14.187981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a651d7-a1c8-4097-a9c6-5690f494f485 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:14.224184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:14.248513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:14.249140 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.249463 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.249733 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] PATH_INFO: `/v3/projects/a4a120e493c4480093217fbdadf469e9` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.250854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4a120e493c4480093217fbdadf469e9)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:14.253457 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:14.373941 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4a120e493c4480093217fbdadf469e9', 'request_id': 'req-e2132038-ec5a-4fed-8df2-a219e481eaed'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:14.379373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4a120e493c4480093217fbdadf469e9', 'request_id': 'req-e2132038-ec5a-4fed-8df2-a219e481eaed'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:14.385572 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4a120e493c4480093217fbdadf469e9', 'request_id': 'req-e2132038-ec5a-4fed-8df2-a219e481eaed'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:14.440286 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] The token cache is being invalidate because project a4a120e493c4480093217fbdadf469e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:14.440579 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:14.441218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-e2132038-ec5a-4fed-8df2-a219e481eaed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:14.442745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1223/2434] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:14 2022] DELETE /identity/v3/projects/a4a120e493c4480093217fbdadf469e9 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:14.447574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e7b0d-ba04-4e22-8ac2-5ced93a17034 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.448052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e7b0d-ba04-4e22-8ac2-5ced93a17034 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.448174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e7b0d-ba04-4e22-8ac2-5ced93a17034 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.468198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e50e7b0d-ba04-4e22-8ac2-5ced93a17034 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:14.469455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e50e7b0d-ba04-4e22-8ac2-5ced93a17034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:14.478710 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:14.529160 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1212/2435] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:14.538900 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f00c839-4928-4c1a-b84c-07ddca638198 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.539190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f00c839-4928-4c1a-b84c-07ddca638198 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.539467 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f00c839-4928-4c1a-b84c-07ddca638198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.560330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8f00c839-4928-4c1a-b84c-07ddca638198 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:14.561612 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8f00c839-4928-4c1a-b84c-07ddca638198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:14.569516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1224/2436] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:14.573586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e7b0d-ba04-4e22-8ac2-5ced93a17034 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:14.583596 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e7b0d-ba04-4e22-8ac2-5ced93a17034 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:14.588165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:14.588769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.589058 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.589376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.590025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:43:14.590331 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:14.591922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:14.604122 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1213/2437] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:43:14.608309 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f00c839-4928-4c1a-b84c-07ddca638198 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:14.618971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f00c839-4928-4c1a-b84c-07ddca638198 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:14.623611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:14.624256 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.624490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.624764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.625419 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:14.627780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:14.716689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1225/2438] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:14 2022] POST /identity/v3/projects => generated 387 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:14.722027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:14.731990 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ac0967-8ce4-470e-b534-381f86ab6f11 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:14.791654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:14.792277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.792564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.792843 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.793565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:14.795560 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:14.864425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1214/2439] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:14 2022] POST /identity/v3/users => generated 294 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:43:14.868819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:14.879816 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a51ed33-6385-4929-b3ff-ce7001917bb0 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:14.884420 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:14.885022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.885307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.885615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.886258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:14.887897 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:14.901667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1226/2440] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:14.902879 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:14.914071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbde39a-4fbc-4544-92e3-6114acc0648f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:14.918779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83877d-0262-452a-91da-6587f4eb87b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:14.919412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83877d-0262-452a-91da-6587f4eb87b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.919700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83877d-0262-452a-91da-6587f4eb87b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.919982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83877d-0262-452a-91da-6587f4eb87b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.920633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d83877d-0262-452a-91da-6587f4eb87b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:14.922267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d83877d-0262-452a-91da-6587f4eb87b8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:14.932987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1215/2441] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:14.936986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:14.947184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b83c129-c79f-4352-bfc5-02a20809bf20 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:14.951668 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:14.952303 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:14.952585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:14.952861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] PATH_INFO: `/v3/projects/0a26c4a146214cf0842155d35ad01622/users/5cf7c3c4e311408bbdd3effb859b472f/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:14.967625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a26c4a146214cf0842155d35ad01622, user_id=5cf7c3c4e311408bbdd3effb859b472f, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:14.970296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.018081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1227/2442] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:14 2022] PUT /identity/v3/projects/0a26c4a146214cf0842155d35ad01622/users/5cf7c3c4e311408bbdd3effb859b472f/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:43:15.028168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc8d7b-4d87-4ae4-ada8-c70f57c21efe None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.028543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc8d7b-4d87-4ae4-ada8-c70f57c21efe None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.028892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc8d7b-4d87-4ae4-ada8-c70f57c21efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.057168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-16dc8d7b-4d87-4ae4-ada8-c70f57c21efe None None] MFA Rules not processed for user `5cf7c3c4e311408bbdd3effb859b472f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:15.058437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-16dc8d7b-4d87-4ae4-ada8-c70f57c21efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:15.123460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1216/2443] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:15 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:15.128540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:15.187974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484ea47-9b9b-41d5-a4a0-1b41ed3f0b21 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:15.193476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:15.194126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.194411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.194686 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.195344 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:15.197279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.202736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:15.288008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1228/2444] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:15 2022] POST /identity/v3/projects => generated 385 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:15.291865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc8d7b-4d87-4ae4-ada8-c70f57c21efe None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:15.302863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc8d7b-4d87-4ae4-ada8-c70f57c21efe None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:15.307443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:15.359184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:15.360086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.360577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.361283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.362014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:15.363966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.423194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1217/2445] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:15 2022] POST /identity/v3/users => generated 299 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:15.427673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:15.438263 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2f490e-d217-4f4a-a239-e0b6de3f986e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:15.443494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:15.444091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.444376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.444657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.445294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:15.446910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.457290 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1229/2446] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:15.461279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:15.471654 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e68bf-8225-4a28-931b-7224a82b618d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:15.476388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:15.477056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.477362 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.477644 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] PATH_INFO: `/v3/projects/cdf82383a89649ca835af6eb38c93edc/users/b5da99dd27ac4e7684b97208639855b0/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.493008 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cdf82383a89649ca835af6eb38c93edc, user_id=b5da99dd27ac4e7684b97208639855b0, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:15.495745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.555296 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1218/2447] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:15 2022] PUT /identity/v3/projects/cdf82383a89649ca835af6eb38c93edc/users/b5da99dd27ac4e7684b97208639855b0/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:15.559238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:15.570203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa04cfdc-ec87-4de1-955b-e2999423aa3e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:15.627851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e533bbee-c217-4731-a3f4-e1d172c0944b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:15.628459 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e533bbee-c217-4731-a3f4-e1d172c0944b admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.628750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e533bbee-c217-4731-a3f4-e1d172c0944b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.629026 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e533bbee-c217-4731-a3f4-e1d172c0944b admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.629697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e533bbee-c217-4731-a3f4-e1d172c0944b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:15.631296 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e533bbee-c217-4731-a3f4-e1d172c0944b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.641089 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1230/2448] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:15 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:15.645243 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:15.655584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb4fc5a-487c-440e-9619-17ba6354444b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:15.660572 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:15.661236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.661550 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.661826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] PATH_INFO: `/v3/projects/cdf82383a89649ca835af6eb38c93edc/users/b5da99dd27ac4e7684b97208639855b0/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.663787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cdf82383a89649ca835af6eb38c93edc, user_id=b5da99dd27ac4e7684b97208639855b0, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:15.666458 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.677467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1219/2449] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:15 2022] PUT /identity/v3/projects/cdf82383a89649ca835af6eb38c93edc/users/b5da99dd27ac4e7684b97208639855b0/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:15.687267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23c933-d7a2-4ef7-92bb-765210e0f8dc None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.687558 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23c933-d7a2-4ef7-92bb-765210e0f8dc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.687841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23c933-d7a2-4ef7-92bb-765210e0f8dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.715803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8c23c933-d7a2-4ef7-92bb-765210e0f8dc None None] MFA Rules not processed for user `b5da99dd27ac4e7684b97208639855b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:15.717011 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8c23c933-d7a2-4ef7-92bb-765210e0f8dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:15.787077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1231/2450] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:15 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:15.798250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:15.800073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:15.856597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e16a1c-9faa-4ee7-8ca3-df0124f39af8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:15.860218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d470e1d-90e2-458f-bb1d-bbc1112ff941 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:15.860819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d470e1d-90e2-458f-bb1d-bbc1112ff941 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:15.861197 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d470e1d-90e2-458f-bb1d-bbc1112ff941 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:15.861470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d470e1d-90e2-458f-bb1d-bbc1112ff941 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:15.870640 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d470e1d-90e2-458f-bb1d-bbc1112ff941 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:15.872059 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d470e1d-90e2-458f-bb1d-bbc1112ff941 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:15.882484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1220/2451] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:15 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:15.965532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c23c933-d7a2-4ef7-92bb-765210e0f8dc None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:16.014069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:16.019348 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c23c933-d7a2-4ef7-92bb-765210e0f8dc None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:16.024022 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:16.028798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:16.028798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:16.028798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] PATH_INFO: `/v3/users/5cf7c3c4e311408bbdd3effb859b472f` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:16.028798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5cf7c3c4e311408bbdd3effb859b472f)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:16.028798 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:16.162405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5cf7c3c4e311408bbdd3effb859b472f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.163079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5cf7c3c4e311408bbdd3effb859b472f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.175540 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5cf7c3c4e311408bbdd3effb859b472f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.182764 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5cf7c3c4e311408bbdd3effb859b472f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.183365 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5cf7c3c4e311408bbdd3effb859b472f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.228535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1232/2452] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:15 2022] DELETE /identity/v3/users/5cf7c3c4e311408bbdd3effb859b472f => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:16.233138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:16.233468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:16.233754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:16.253913 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:16.255116 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:16.312865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1221/2453] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:16.322577 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:16.324005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:16.383539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb63af-d16b-4ad5-97ac-e65e831162c6 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:16.387222 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:16.387827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:16.388110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:16.388388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:16.397487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:16.398915 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:16.408590 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1233/2454] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:43:16.802548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:16.804127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:16.814502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:16.818038 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7a820-072c-4362-97de-7b60c8176005 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:16.822807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:16.823453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:16.823738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:16.824016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] PATH_INFO: `/v3/projects/0a26c4a146214cf0842155d35ad01622` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:16.825184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a26c4a146214cf0842155d35ad01622)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:16.827140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:16.921288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a26c4a146214cf0842155d35ad01622', 'request_id': 'req-bd40eee9-091e-48ee-bab2-a3d616cf9f20'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.971586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a26c4a146214cf0842155d35ad01622', 'request_id': 'req-bd40eee9-091e-48ee-bab2-a3d616cf9f20'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.978609 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a26c4a146214cf0842155d35ad01622', 'request_id': 'req-bd40eee9-091e-48ee-bab2-a3d616cf9f20'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.990488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] The token cache is being invalidate because project 0a26c4a146214cf0842155d35ad01622 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:16.990785 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.991447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:16.993223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1222/2455] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:16 2022] DELETE /identity/v3/projects/0a26c4a146214cf0842155d35ad01622 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:16.997353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:17.007946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75863300-94a8-4fbe-b478-215a7814c538 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:17.064527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:17.065090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:17.065401 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:17.065682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] PATH_INFO: `/v3/users/b5da99dd27ac4e7684b97208639855b0` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:17.067364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5da99dd27ac4e7684b97208639855b0)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:17.069283 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:17.090067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:17.243326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5da99dd27ac4e7684b97208639855b0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:17.243967 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5da99dd27ac4e7684b97208639855b0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:17.256148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5da99dd27ac4e7684b97208639855b0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:17.263097 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5da99dd27ac4e7684b97208639855b0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:17.263738 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-a1c43107-7f95-4fda-b33f-f8f8c19fe101 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5da99dd27ac4e7684b97208639855b0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:17.337194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1234/2456] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:16 2022] DELETE /identity/v3/users/b5da99dd27ac4e7684b97208639855b0 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:17.727491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:17.738702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:17.744143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd40eee9-091e-48ee-bab2-a3d616cf9f20 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:17.804912 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:17.805608 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:17.805856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:17.806133 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] PATH_INFO: `/v3/projects/cdf82383a89649ca835af6eb38c93edc` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:17.807281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cdf82383a89649ca835af6eb38c93edc)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:17.809191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:17.920145 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cdf82383a89649ca835af6eb38c93edc', 'request_id': 'req-5c8fe531-eb8c-4628-81cf-235594ab9521'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:18.011702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cdf82383a89649ca835af6eb38c93edc', 'request_id': 'req-5c8fe531-eb8c-4628-81cf-235594ab9521'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:18.018698 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cdf82383a89649ca835af6eb38c93edc', 'request_id': 'req-5c8fe531-eb8c-4628-81cf-235594ab9521'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:18.029618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] The token cache is being invalidate because project cdf82383a89649ca835af6eb38c93edc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:18.030026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:18.030775 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c8fe531-eb8c-4628-81cf-235594ab9521 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:18.032261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1223/2457] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:17 2022] DELETE /identity/v3/projects/cdf82383a89649ca835af6eb38c93edc => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:18.037125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012ecaa-9e01-4eb3-966e-a4d3386d1e7e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.037232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012ecaa-9e01-4eb3-966e-a4d3386d1e7e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.038031 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012ecaa-9e01-4eb3-966e-a4d3386d1e7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.058906 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a012ecaa-9e01-4eb3-966e-a4d3386d1e7e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:18.060122 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a012ecaa-9e01-4eb3-966e-a4d3386d1e7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:18.121079 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1235/2458] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:18.129761 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a40ae-d4f4-40cc-8ae4-166bb65f7e68 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.130052 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a40ae-d4f4-40cc-8ae4-166bb65f7e68 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.130331 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a40ae-d4f4-40cc-8ae4-166bb65f7e68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.149677 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9d8a40ae-d4f4-40cc-8ae4-166bb65f7e68 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:18.150865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9d8a40ae-d4f4-40cc-8ae4-166bb65f7e68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:18.158480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1224/2459] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:18.163297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012ecaa-9e01-4eb3-966e-a4d3386d1e7e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.173069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012ecaa-9e01-4eb3-966e-a4d3386d1e7e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.177646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.178890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.179174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.179468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.180110 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:43:18.180421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.182082 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:18.193376 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1236/2460] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:43:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:43:18.197428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8a40ae-d4f4-40cc-8ae4-166bb65f7e68 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.206683 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8a40ae-d4f4-40cc-8ae4-166bb65f7e68 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.211335 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.211935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.212221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.212506 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.213143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.215092 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:18.224662 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:18.276151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1225/2461] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:18 2022] POST /identity/v3/projects => generated 381 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:18.280735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.291782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e2dba7-608d-4168-ae4b-b781aca4e26e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.347930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.348527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.348813 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.349080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.349723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.352218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:18.423003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1237/2462] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:18 2022] POST /identity/v3/users => generated 291 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:18.427257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.437667 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a1da72-8d9e-4e9a-8261-a96d4177cf19 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.442582 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.443297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.443692 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.444010 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.444724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.446504 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:18.456905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1226/2463] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:43:18.462044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.472170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907219e7-454a-416c-b5e4-c2083137d03c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.477775 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29dd4e5-ff6f-409f-860e-ba0507630f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.478379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29dd4e5-ff6f-409f-860e-ba0507630f0c admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.478664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29dd4e5-ff6f-409f-860e-ba0507630f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.478944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29dd4e5-ff6f-409f-860e-ba0507630f0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.479582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29dd4e5-ff6f-409f-860e-ba0507630f0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.481180 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29dd4e5-ff6f-409f-860e-ba0507630f0c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:18.492308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1238/2464] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:18.496487 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.506378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e96b61-b93d-4f14-ad5f-3b5e28a84d33 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.510976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.511628 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.511911 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.512188 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] PATH_INFO: `/v3/projects/4a0cf70518d94484b37968e1438825cf/users/b6ed4ae396d043d590b7fda560500c5b/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.528230 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a0cf70518d94484b37968e1438825cf, user_id=b6ed4ae396d043d590b7fda560500c5b, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.530916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:18.594198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1227/2465] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:18 2022] PUT /identity/v3/projects/4a0cf70518d94484b37968e1438825cf/users/b6ed4ae396d043d590b7fda560500c5b/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:18.605405 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab757c3a-72d5-4460-a86f-ac62698150d3 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.605693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab757c3a-72d5-4460-a86f-ac62698150d3 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.605969 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab757c3a-72d5-4460-a86f-ac62698150d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.614305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:18.633952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ab757c3a-72d5-4460-a86f-ac62698150d3 None None] MFA Rules not processed for user `b6ed4ae396d043d590b7fda560500c5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:18.635170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ab757c3a-72d5-4460-a86f-ac62698150d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:18.699690 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1239/2466] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:18 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:43:18.704221 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.766040 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b091d2-566f-45a6-bb73-10e3557c2b52 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.770693 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.771302 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.771587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.772497 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.773127 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.775073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:18.875873 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1228/2467] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:18 2022] POST /identity/v3/projects => generated 379 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:18.880060 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab757c3a-72d5-4460-a86f-ac62698150d3 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:18.892252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab757c3a-72d5-4460-a86f-ac62698150d3 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:18.951727 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:18.952269 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:18.952555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:18.952834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:18.953506 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:18.955425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:19.017166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1240/2468] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:18 2022] POST /identity/v3/users => generated 296 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:19.021254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:19.032514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879ef632-c9b1-40a5-b2a3-94a23b66e9bc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:19.037053 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:19.037803 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:19.038283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:19.038549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:19.039236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:19.040832 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:19.051294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1229/2469] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:19.055504 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:19.065464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91925ec4-f0ea-42e4-9fad-f05da321d165 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:19.071039 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:19.071694 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:19.071980 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:19.072258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] PATH_INFO: `/v3/projects/aae8a86cff6746c4a3f87006ea477437/users/f705373d04354e63baee9e96472991f7/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:19.087519 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aae8a86cff6746c4a3f87006ea477437, user_id=f705373d04354e63baee9e96472991f7, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:19.090194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:19.166371 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1241/2470] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:19 2022] PUT /identity/v3/projects/aae8a86cff6746c4a3f87006ea477437/users/f705373d04354e63baee9e96472991f7/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:19.170234 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:19.180793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6c3405-6116-4fda-83d5-34c3598e636e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:19.236728 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1558312-26dd-473d-aeba-7c229943cc6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:19.237233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1558312-26dd-473d-aeba-7c229943cc6b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:19.237552 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1558312-26dd-473d-aeba-7c229943cc6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:19.237852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1558312-26dd-473d-aeba-7c229943cc6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:19.238472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1558312-26dd-473d-aeba-7c229943cc6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:19.240062 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1558312-26dd-473d-aeba-7c229943cc6b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:19.250673 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1230/2471] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:19.254851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:19.265839 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc60dadd-75a5-41a0-ad1c-56610892da28 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:19.270391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:19.271044 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:19.271305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:19.271574 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] PATH_INFO: `/v3/projects/aae8a86cff6746c4a3f87006ea477437/users/f705373d04354e63baee9e96472991f7/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:19.273476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aae8a86cff6746c4a3f87006ea477437, user_id=f705373d04354e63baee9e96472991f7, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:19.276102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:19.286117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1242/2472] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:19 2022] PUT /identity/v3/projects/aae8a86cff6746c4a3f87006ea477437/users/f705373d04354e63baee9e96472991f7/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:19.297153 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea71092-30a2-4249-93d8-d411be48de43 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:19.297474 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea71092-30a2-4249-93d8-d411be48de43 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:19.297752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea71092-30a2-4249-93d8-d411be48de43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:19.326348 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-dea71092-30a2-4249-93d8-d411be48de43 None None] MFA Rules not processed for user `f705373d04354e63baee9e96472991f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:19.327548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dea71092-30a2-4249-93d8-d411be48de43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:19.339437 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:19.396854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1231/2473] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:19 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:19.407441 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:19.409152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:19.462952 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:19.468137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f57643-83ad-4a46-b303-21e8cd8ef5c1 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:19.471894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3016f79-6788-4b41-9770-610326613c20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:19.472388 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3016f79-6788-4b41-9770-610326613c20 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:19.472665 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3016f79-6788-4b41-9770-610326613c20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:19.472937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3016f79-6788-4b41-9770-610326613c20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:19.482757 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3016f79-6788-4b41-9770-610326613c20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:19.484186 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3016f79-6788-4b41-9770-610326613c20 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:19.494270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1243/2474] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:19 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:19.947549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea71092-30a2-4249-93d8-d411be48de43 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:20.007590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea71092-30a2-4249-93d8-d411be48de43 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:20.012779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:20.013378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:20.013658 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:20.013936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] PATH_INFO: `/v3/users/b6ed4ae396d043d590b7fda560500c5b` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:20.015055 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6ed4ae396d043d590b7fda560500c5b)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:20.016977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:20.192225 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6ed4ae396d043d590b7fda560500c5b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:20.204664 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6ed4ae396d043d590b7fda560500c5b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:20.268825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6ed4ae396d043d590b7fda560500c5b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:20.269415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6ed4ae396d043d590b7fda560500c5b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:20.270036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6ed4ae396d043d590b7fda560500c5b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:20.282686 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1232/2475] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:19 2022] DELETE /identity/v3/users/b6ed4ae396d043d590b7fda560500c5b => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:20.287310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:20.287787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:20.287880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:20.307702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:20.308909 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:20.365719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1244/2476] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:20.376178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:20.377595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:20.408971 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:20.434555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfd1d59-c624-493b-9450-70e39fe89aab admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:20.438191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:20.438802 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:20.439106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:20.439591 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:20.449414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:20.454949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:20.460220 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1233/2477] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:20.888083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:20.889552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:20.901986 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca1946f-309e-49e8-acb4-301461860468 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:20.907175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:20.907815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:20.908103 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:20.908379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] PATH_INFO: `/v3/projects/4a0cf70518d94484b37968e1438825cf` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:20.909529 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a0cf70518d94484b37968e1438825cf)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:20.911443 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:20.996282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a0cf70518d94484b37968e1438825cf', 'request_id': 'req-800dbea1-a2b2-4c06-a58c-96abc482253b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.001874 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a0cf70518d94484b37968e1438825cf', 'request_id': 'req-800dbea1-a2b2-4c06-a58c-96abc482253b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.008169 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a0cf70518d94484b37968e1438825cf', 'request_id': 'req-800dbea1-a2b2-4c06-a58c-96abc482253b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.073774 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] The token cache is being invalidate because project 4a0cf70518d94484b37968e1438825cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:21.074072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.074703 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.076234 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1245/2478] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:20 2022] DELETE /identity/v3/projects/4a0cf70518d94484b37968e1438825cf => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:21.080454 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:21.090269 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064fb7cc-61da-45a2-986c-43a69b2a3286 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:21.145306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:21.145894 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:21.146174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:21.146447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] PATH_INFO: `/v3/users/f705373d04354e63baee9e96472991f7` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:21.147547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f705373d04354e63baee9e96472991f7)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:21.149536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:21.310765 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f705373d04354e63baee9e96472991f7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.322594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f705373d04354e63baee9e96472991f7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.394378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f705373d04354e63baee9e96472991f7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.394994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f705373d04354e63baee9e96472991f7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.395652 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ac68eb46-dce8-4d2b-8347-c176baba28f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f705373d04354e63baee9e96472991f7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:21.408528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1234/2479] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:21 2022] DELETE /identity/v3/users/f705373d04354e63baee9e96472991f7 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:21.813532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:21.815357 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:21.830937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800dbea1-a2b2-4c06-a58c-96abc482253b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:21.888310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:21.888935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:21.889230 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:21.889524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] PATH_INFO: `/v3/projects/aae8a86cff6746c4a3f87006ea477437` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:21.890635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] RBAC: Authorizing `identity:delete_project(project_id=aae8a86cff6746c4a3f87006ea477437)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:21.892539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:21.908962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:22.024087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aae8a86cff6746c4a3f87006ea477437', 'request_id': 'req-0f9fc735-8f81-4207-ac00-13994cc0d81e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:22.030098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aae8a86cff6746c4a3f87006ea477437', 'request_id': 'req-0f9fc735-8f81-4207-ac00-13994cc0d81e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:22.036521 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aae8a86cff6746c4a3f87006ea477437', 'request_id': 'req-0f9fc735-8f81-4207-ac00-13994cc0d81e'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:22.101997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] The token cache is being invalidate because project aae8a86cff6746c4a3f87006ea477437 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:22.102318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:22.103025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-0f9fc735-8f81-4207-ac00-13994cc0d81e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:22.104507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1246/2480] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:21 2022] DELETE /identity/v3/projects/aae8a86cff6746c4a3f87006ea477437 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:22.109525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b23bce-c4fb-4e31-b339-3a80cd448311 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.109987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b23bce-c4fb-4e31-b339-3a80cd448311 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.110511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b23bce-c4fb-4e31-b339-3a80cd448311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.130930 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-26b23bce-c4fb-4e31-b339-3a80cd448311 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:22.132136 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-26b23bce-c4fb-4e31-b339-3a80cd448311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:22.177546 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:22.191712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1235/2481] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:22.200244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b07b89-a707-4818-869f-2abf66049042 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.200528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b07b89-a707-4818-869f-2abf66049042 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.200800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b07b89-a707-4818-869f-2abf66049042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.220993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b8b07b89-a707-4818-869f-2abf66049042 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:22.228171 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b8b07b89-a707-4818-869f-2abf66049042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:22.230634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1247/2482] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:43:22.234892 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b23bce-c4fb-4e31-b339-3a80cd448311 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:22.246005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b23bce-c4fb-4e31-b339-3a80cd448311 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:22.250518 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:22.251144 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.251461 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.251739 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.252379 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:43:22.252679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:22.254288 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:22.266865 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1236/2483] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:43:22.271778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b07b89-a707-4818-869f-2abf66049042 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:22.281312 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b07b89-a707-4818-869f-2abf66049042 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:22.285965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:22.286562 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.286844 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.287117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.287755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:22.289706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:22.373868 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1248/2484] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:22 2022] POST /identity/v3/projects => generated 375 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:22.378081 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:22.388916 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e151ed1-af03-46f5-9196-ee5c1a4e49ee admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:22.444297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:22.444901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.445182 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.445503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.446123 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:22.448223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:22.508045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1237/2485] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:22 2022] POST /identity/v3/users => generated 288 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:22.512945 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:22.523337 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b9316c-fd20-431a-b82e-9d1bfc57010c admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:22.528464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:22.529061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.529381 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.529850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.530509 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:22.532102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:22.542051 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1249/2486] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:43:22.545997 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:22.555746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db0a7ee-2afe-4c46-8fe3-401259036867 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:22.560294 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778ab7b9-a101-49b9-b22f-0a13cb9b3042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:22.560880 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ab7b9-a101-49b9-b22f-0a13cb9b3042 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.561163 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ab7b9-a101-49b9-b22f-0a13cb9b3042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.561468 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ab7b9-a101-49b9-b22f-0a13cb9b3042 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.562098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778ab7b9-a101-49b9-b22f-0a13cb9b3042 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:22.563671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778ab7b9-a101-49b9-b22f-0a13cb9b3042 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:22.574579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1238/2487] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:22.578706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:22.589265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e777383-ec46-45b0-b203-f094a1920ca7 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:22.593992 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:22.594641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.594924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.595227 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] PATH_INFO: `/v3/projects/cb34bb0822b544bcad3f029ad9e26293/users/9c2dd0e373194de69e9ef58c034d2f48/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.610475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb34bb0822b544bcad3f029ad9e26293, user_id=9c2dd0e373194de69e9ef58c034d2f48, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:22.613128 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:22.691652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1250/2488] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:22 2022] PUT /identity/v3/projects/cb34bb0822b544bcad3f029ad9e26293/users/9c2dd0e373194de69e9ef58c034d2f48/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:22.702039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6178c66-248f-4e8c-a041-97d85e71a2d1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.702265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6178c66-248f-4e8c-a041-97d85e71a2d1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.702544 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6178c66-248f-4e8c-a041-97d85e71a2d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.730554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e6178c66-248f-4e8c-a041-97d85e71a2d1 None None] MFA Rules not processed for user `9c2dd0e373194de69e9ef58c034d2f48`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:22.731748 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e6178c66-248f-4e8c-a041-97d85e71a2d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:22.795769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1239/2489] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:22.806489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:22.808170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:22.863963 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83805a93-15cc-4ebb-be6b-a4de3d66ecbe admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:22.868246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b63bef-4317-40c3-a998-f248cc3b93b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:22.868948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b63bef-4317-40c3-a998-f248cc3b93b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:22.869236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b63bef-4317-40c3-a998-f248cc3b93b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:22.869640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b63bef-4317-40c3-a998-f248cc3b93b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:22.878145 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b63bef-4317-40c3-a998-f248cc3b93b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:22.879556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b63bef-4317-40c3-a998-f248cc3b93b6 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:22.889172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1251/2490] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:26.863672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6178c66-248f-4e8c-a041-97d85e71a2d1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:26.910812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:26.927011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6178c66-248f-4e8c-a041-97d85e71a2d1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:26.932603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:26.933242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:26.933564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:26.933848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] PATH_INFO: `/v3/users/9c2dd0e373194de69e9ef58c034d2f48` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:26.934974 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c2dd0e373194de69e9ef58c034d2f48)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:26.936896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:27.128481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c2dd0e373194de69e9ef58c034d2f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:27.140642 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c2dd0e373194de69e9ef58c034d2f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:27.233583 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c2dd0e373194de69e9ef58c034d2f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:27.234193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c2dd0e373194de69e9ef58c034d2f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:27.234794 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c2dd0e373194de69e9ef58c034d2f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:27.248313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1240/2491] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:26 2022] DELETE /identity/v3/users/9c2dd0e373194de69e9ef58c034d2f48 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:27.252890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:27.253389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:27.253524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:27.272799 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:27.274084 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:27.313299 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:27.333596 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1252/2492] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:27.343731 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:27.345124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:27.405150 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac40fbc-338d-4fcb-8d34-9d68fdf30121 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:27.408862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323e4665-fd36-4a7f-a0cd-d6971e625a9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:27.409577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e4665-fd36-4a7f-a0cd-d6971e625a9c service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:27.409863 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e4665-fd36-4a7f-a0cd-d6971e625a9c service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:27.410143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e4665-fd36-4a7f-a0cd-d6971e625a9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:27.420339 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323e4665-fd36-4a7f-a0cd-d6971e625a9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:27.421812 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323e4665-fd36-4a7f-a0cd-d6971e625a9c service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:27.431271 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1241/2493] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:27.801167 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:27.802630 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:27.816379 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ae9184-d46c-4753-a0de-28347e0b3237 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:27.821052 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:27.821715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:27.822067 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:27.833190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] PATH_INFO: `/v3/projects/cb34bb0822b544bcad3f029ad9e26293` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:27.833190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb34bb0822b544bcad3f029ad9e26293)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:27.833190 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:27.937505 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb34bb0822b544bcad3f029ad9e26293', 'request_id': 'req-66976f18-c27b-43c6-99a3-c8ed52b16985'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:27.943005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb34bb0822b544bcad3f029ad9e26293', 'request_id': 'req-66976f18-c27b-43c6-99a3-c8ed52b16985'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:27.950019 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb34bb0822b544bcad3f029ad9e26293', 'request_id': 'req-66976f18-c27b-43c6-99a3-c8ed52b16985'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:28.033203 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] The token cache is being invalidate because project cb34bb0822b544bcad3f029ad9e26293 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:28.033539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:28.034173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-66976f18-c27b-43c6-99a3-c8ed52b16985 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:28.035674 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1253/2494] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:27 2022] DELETE /identity/v3/projects/cb34bb0822b544bcad3f029ad9e26293 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:28.041313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85efb01-7183-4af9-8e6e-3501ebfe76f5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.041788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85efb01-7183-4af9-8e6e-3501ebfe76f5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.041901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85efb01-7183-4af9-8e6e-3501ebfe76f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.062387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a85efb01-7183-4af9-8e6e-3501ebfe76f5 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:28.063576 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a85efb01-7183-4af9-8e6e-3501ebfe76f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:28.125931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1242/2495] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:28.134185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd3d172-3a52-4c71-8e13-02a81a82b610 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.137802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd3d172-3a52-4c71-8e13-02a81a82b610 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.137802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd3d172-3a52-4c71-8e13-02a81a82b610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.156126 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-8fd3d172-3a52-4c71-8e13-02a81a82b610 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:28.157383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-8fd3d172-3a52-4c71-8e13-02a81a82b610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:28.165535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1254/2496] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:43:28.169714 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85efb01-7183-4af9-8e6e-3501ebfe76f5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:28.179442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85efb01-7183-4af9-8e6e-3501ebfe76f5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:28.184517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:28.185112 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.185422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.185706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.186373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:43:28.186676 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:28.188309 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:28.190555 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:28.200292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1243/2497] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:43:28.204494 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd3d172-3a52-4c71-8e13-02a81a82b610 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:28.213711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd3d172-3a52-4c71-8e13-02a81a82b610 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:28.218561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:28.219158 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.219440 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.219711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.220343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:28.222311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:28.286260 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1255/2498] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:28 2022] POST /identity/v3/projects => generated 375 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:28.290502 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:28.300376 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6877467f-48fa-42d7-8a94-2be41952704c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:28.357408 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:28.358007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.358292 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.358568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.359195 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:28.361102 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:28.440203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1244/2499] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:28 2022] POST /identity/v3/users => generated 288 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:43:28.444421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:28.458791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd69663-31ae-431e-bb09-e9ae53915d7b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:28.463719 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:28.464469 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.464730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.465005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.465743 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:28.467385 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:28.477212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1256/2500] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:28.481976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:28.491807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd715a81-0728-43dc-9ac8-2f21e0035b87 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:28.497419 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983fb611-1165-4309-8c5f-48484b812707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:28.498016 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983fb611-1165-4309-8c5f-48484b812707 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.498304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983fb611-1165-4309-8c5f-48484b812707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.498581 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983fb611-1165-4309-8c5f-48484b812707 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.499227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983fb611-1165-4309-8c5f-48484b812707 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:28.500826 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983fb611-1165-4309-8c5f-48484b812707 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:28.511291 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1245/2501] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:28.515265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:28.525091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e063e1c3-6868-4ba2-bc5e-be8b46ae9845 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:28.529997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:28.530645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.530929 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.531206 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] PATH_INFO: `/v3/projects/2d2826b7e8554d23ba6926703e8cbd8d/users/c1af604e2413488da9cb89baef9ec855/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.546257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d2826b7e8554d23ba6926703e8cbd8d, user_id=c1af604e2413488da9cb89baef9ec855, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:28.548896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:28.623534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1257/2502] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:28 2022] PUT /identity/v3/projects/2d2826b7e8554d23ba6926703e8cbd8d/users/c1af604e2413488da9cb89baef9ec855/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:28.633809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccf22d7-3a39-4a2a-a8e9-58611108d7fc None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.634425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccf22d7-3a39-4a2a-a8e9-58611108d7fc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.634425 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccf22d7-3a39-4a2a-a8e9-58611108d7fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.663763 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3ccf22d7-3a39-4a2a-a8e9-58611108d7fc None None] MFA Rules not processed for user `c1af604e2413488da9cb89baef9ec855`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:28.665087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3ccf22d7-3a39-4a2a-a8e9-58611108d7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:28.731226 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1246/2503] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:28.741567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:28.743541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:28.799955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca87991-a34b-4ef8-aeef-357746904ad7 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:28.803591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dcf149-0c13-4b60-bea7-006fba3647eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:28.804193 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dcf149-0c13-4b60-bea7-006fba3647eb service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:28.804475 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dcf149-0c13-4b60-bea7-006fba3647eb service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:28.804750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dcf149-0c13-4b60-bea7-006fba3647eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:28.813009 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dcf149-0c13-4b60-bea7-006fba3647eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:28.814469 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dcf149-0c13-4b60-bea7-006fba3647eb service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:28.823099 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1258/2504] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:38.727674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=68,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:38.819153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:40.066383 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccf22d7-3a39-4a2a-a8e9-58611108d7fc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:40.128641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccf22d7-3a39-4a2a-a8e9-58611108d7fc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:40.133178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:40.133835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:40.134121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:40.134400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] PATH_INFO: `/v3/users/c1af604e2413488da9cb89baef9ec855` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:40.135509 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1af604e2413488da9cb89baef9ec855)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:40.138156 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:40.269299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1af604e2413488da9cb89baef9ec855'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:40.281445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1af604e2413488da9cb89baef9ec855'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:40.329899 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1af604e2413488da9cb89baef9ec855'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:40.330551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1af604e2413488da9cb89baef9ec855'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:40.331154 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1af604e2413488da9cb89baef9ec855'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:40.345381 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1247/2505] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:43:40 2022] DELETE /identity/v3/users/c1af604e2413488da9cb89baef9ec855 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:40.349567 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:40.350023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:40.350146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:40.369012 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:40.370232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:40.429446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1259/2506] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:40.439531 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:40.440922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:40.498152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f49981a-651b-4c35-b5dd-234c22b887f8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:40.501808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1bda88-a146-4eb3-8f64-f2abdc8a6b90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:40.502411 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1bda88-a146-4eb3-8f64-f2abdc8a6b90 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:40.502696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1bda88-a146-4eb3-8f64-f2abdc8a6b90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:40.502975 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1bda88-a146-4eb3-8f64-f2abdc8a6b90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:40.511877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1bda88-a146-4eb3-8f64-f2abdc8a6b90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:40.513976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1bda88-a146-4eb3-8f64-f2abdc8a6b90 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:40.523191 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1248/2507] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:40.909010 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:40.910759 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:40.922993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b51e3e6-1502-4795-8d1c-45413d786e09 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:40.927789 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:40.928482 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:40.928745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:40.929017 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] PATH_INFO: `/v3/projects/2d2826b7e8554d23ba6926703e8cbd8d` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:40.930237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d2826b7e8554d23ba6926703e8cbd8d)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:40.932970 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.035151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d2826b7e8554d23ba6926703e8cbd8d', 'request_id': 'req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:41.040819 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d2826b7e8554d23ba6926703e8cbd8d', 'request_id': 'req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:41.047334 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d2826b7e8554d23ba6926703e8cbd8d', 'request_id': 'req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:41.118928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] The token cache is being invalidate because project 2d2826b7e8554d23ba6926703e8cbd8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:43:41.119217 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:41.119834 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-7404bec5-9f9a-497a-bef1-2f6c5709ee7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:43:41.121404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1260/2508] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:43:40 2022] DELETE /identity/v3/projects/2d2826b7e8554d23ba6926703e8cbd8d => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:41.126338 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaad7e0-ec4c-4553-a08b-95968d2b189f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.126844 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaad7e0-ec4c-4553-a08b-95968d2b189f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.126941 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaad7e0-ec4c-4553-a08b-95968d2b189f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.147373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3eaad7e0-ec4c-4553-a08b-95968d2b189f None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:41.148574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3eaad7e0-ec4c-4553-a08b-95968d2b189f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:41.165048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:41.206793 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1249/2509] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:41.215709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab682afc-1ae8-4d39-9426-fdcdbf8519d0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.216005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab682afc-1ae8-4d39-9426-fdcdbf8519d0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.216284 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab682afc-1ae8-4d39-9426-fdcdbf8519d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.236552 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ab682afc-1ae8-4d39-9426-fdcdbf8519d0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:41.237788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ab682afc-1ae8-4d39-9426-fdcdbf8519d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:41.246763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1261/2510] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:43:41.250927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaad7e0-ec4c-4553-a08b-95968d2b189f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:41.260447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaad7e0-ec4c-4553-a08b-95968d2b189f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:41.264988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:41.265632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.265908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.266184 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.266827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:43:41.267132 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:41.268712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.280545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1250/2511] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:43:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 20:43:41.284728 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab682afc-1ae8-4d39-9426-fdcdbf8519d0 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:41.294477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab682afc-1ae8-4d39-9426-fdcdbf8519d0 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:41.299124 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:41.299737 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.300007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.300282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.301550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:41.305164 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.372960 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1262/2512] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:41 2022] POST /identity/v3/projects => generated 373 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:41.377697 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:41.387249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a812e9c-f27e-4435-8a0a-abdf26f535f4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:41.448851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:41.449501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.449786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.450060 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.450687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:41.452586 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.535364 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1251/2513] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:41 2022] POST /identity/v3/users => generated 287 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:41.539638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:41.550023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6922992-2375-47ec-af80-346ea2d155bc admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:41.555675 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:41.556275 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.556559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.556838 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.557620 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:41.559211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.570210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1263/2514] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:41.574031 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:41.584048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d06bc-c2e1-4930-95d2-14133cdbec6c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:41.588786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640b2774-fd83-40a4-a2da-953de613c302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:41.589415 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640b2774-fd83-40a4-a2da-953de613c302 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.589706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640b2774-fd83-40a4-a2da-953de613c302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.589984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640b2774-fd83-40a4-a2da-953de613c302 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.590626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640b2774-fd83-40a4-a2da-953de613c302 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:41.592215 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640b2774-fd83-40a4-a2da-953de613c302 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.602672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1252/2515] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:41 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:41.606635 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:41.617770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f20186-15b3-4e52-b6c9-dcd79614026b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:41.622446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04acd125-befe-413c-8007-0f909171872a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:41.623129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04acd125-befe-413c-8007-0f909171872a admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.623412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04acd125-befe-413c-8007-0f909171872a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.623698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04acd125-befe-413c-8007-0f909171872a admin admin] PATH_INFO: `/v3/projects/2d9ea1131d80461da67996b2dc844f49/users/d41ea52c88f1436188152eab37f47ab6/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.639081 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04acd125-befe-413c-8007-0f909171872a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d9ea1131d80461da67996b2dc844f49, user_id=d41ea52c88f1436188152eab37f47ab6, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:41.641776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04acd125-befe-413c-8007-0f909171872a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.706542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1264/2516] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:41 2022] PUT /identity/v3/projects/2d9ea1131d80461da67996b2dc844f49/users/d41ea52c88f1436188152eab37f47ab6/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:41.711788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f06e25-d816-41f3-b2c5-3d7bac69b2eb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.712098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f06e25-d816-41f3-b2c5-3d7bac69b2eb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.712402 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f06e25-d816-41f3-b2c5-3d7bac69b2eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.731079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-46f06e25-d816-41f3-b2c5-3d7bac69b2eb None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:41.732333 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-46f06e25-d816-41f3-b2c5-3d7bac69b2eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:41.791289 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1253/2517] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:41.800805 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04acd125-befe-413c-8007-0f909171872a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:41.803224 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-04acd125-befe-413c-8007-0f909171872a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:41.825811 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:41.860532 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04acd125-befe-413c-8007-0f909171872a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:41.865264 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:41.865901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:41.866445 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:41.866723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:41.875717 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:41.877153 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:41.897974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1265/2518] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:47.354663 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57896286-4066-419d-a748-3efa31f8c9ae None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:47.355242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57896286-4066-419d-a748-3efa31f8c9ae None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:47.355242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57896286-4066-419d-a748-3efa31f8c9ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:47.389709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-57896286-4066-419d-a748-3efa31f8c9ae None None] MFA Rules not processed for user `d41ea52c88f1436188152eab37f47ab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:47.390927 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-57896286-4066-419d-a748-3efa31f8c9ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:47.409363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:47.461199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1254/2519] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:47.465848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:47.476091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac788b5e-26cd-48f8-9fe6-c25590023a31 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:47.481085 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:47.481735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:47.482020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:47.482294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:47.482926 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:47.484848 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:47.561175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1266/2520] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:43:47 2022] POST /identity/v3/projects => generated 371 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:47.566429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57896286-4066-419d-a748-3efa31f8c9ae None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:47.578414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57896286-4066-419d-a748-3efa31f8c9ae None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:47.640205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:47.640830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:47.641111 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:47.641561 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:47.642201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:47.644248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:47.731140 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1255/2521] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:43:47 2022] POST /identity/v3/users => generated 292 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:43:47.735078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:47.747133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e27b163-a4b6-4d42-b015-94dc0493ea1e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:47.751913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:47.752541 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:47.752827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:47.753174 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:47.753892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:47.755866 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:47.769184 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1267/2522] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:47 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:47.773950 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:47.784239 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449ff895-bd26-4c4d-8d3f-8010ba700df1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:47.790047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:47.790719 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:47.791005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:47.791286 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] PATH_INFO: `/v3/projects/a7b034dd1630433fa3d91432072aa33f/users/715cfa404fcc42688a0b28db32c480d0/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:47.806767 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7b034dd1630433fa3d91432072aa33f, user_id=715cfa404fcc42688a0b28db32c480d0, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:47.809453 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:47.889882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1256/2523] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:47 2022] PUT /identity/v3/projects/a7b034dd1630433fa3d91432072aa33f/users/715cfa404fcc42688a0b28db32c480d0/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:47.893912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:47.905242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4cada7-66b7-4a20-97d7-dcf9f81f42bf admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:47.975659 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9b438e-f845-41e9-a6c2-efe6ab6935fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:47.976266 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b438e-f845-41e9-a6c2-efe6ab6935fa admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:47.976548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b438e-f845-41e9-a6c2-efe6ab6935fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:47.976823 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b438e-f845-41e9-a6c2-efe6ab6935fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:47.977499 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9b438e-f845-41e9-a6c2-efe6ab6935fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:47.979102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9b438e-f845-41e9-a6c2-efe6ab6935fa admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:47.988853 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1268/2524] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:43:47 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:43:47.993108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:48.003373 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8512058-b223-4ec8-818e-88d7ccfaefe3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:48.008480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:48.009128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:48.009443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:48.009724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] PATH_INFO: `/v3/projects/a7b034dd1630433fa3d91432072aa33f/users/715cfa404fcc42688a0b28db32c480d0/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:48.011632 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7b034dd1630433fa3d91432072aa33f, user_id=715cfa404fcc42688a0b28db32c480d0, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:48.014326 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:48.025937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1257/2525] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:43:47 2022] PUT /identity/v3/projects/a7b034dd1630433fa3d91432072aa33f/users/715cfa404fcc42688a0b28db32c480d0/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:43:53.931641 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cfcb98-53aa-46d3-b234-9d53c49a10e5 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:53.932127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cfcb98-53aa-46d3-b234-9d53c49a10e5 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:53.932223 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cfcb98-53aa-46d3-b234-9d53c49a10e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:53.959528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-12cfcb98-53aa-46d3-b234-9d53c49a10e5 None None] MFA Rules not processed for user `715cfa404fcc42688a0b28db32c480d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:53.960723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-12cfcb98-53aa-46d3-b234-9d53c49a10e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:54.036004 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1269/2526] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:53 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:43:54.050104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:43:54.053086 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:54.135852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a592b886-d6ff-4397-aa19-82783de95b2f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:43:54.141455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:43:54.142076 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:54.142356 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:54.142647 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:54.148190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:43:54.204174 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:43:54.205841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:43:54.215968 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1258/2527] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:43:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:43:56.622176 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978586b-11b9-47d0-8d3d-333d6e1af755 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:43:56.622722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978586b-11b9-47d0-8d3d-333d6e1af755 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:43:56.622722 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978586b-11b9-47d0-8d3d-333d6e1af755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:43:56.643922 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a978586b-11b9-47d0-8d3d-333d6e1af755 None None] MFA Rules not processed for user `d41ea52c88f1436188152eab37f47ab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:43:56.645464 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a978586b-11b9-47d0-8d3d-333d6e1af755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:43:56.653215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1270/2528] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:43:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:44:04.211594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=24 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:44:06.637065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:44:12.257787 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:44:12.259187 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:44:12.319303 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95a2ad0-970c-4d6a-924f-95e7e60bb420 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:44:12.322954 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:44:12.323566 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:44:12.323850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:44:12.324126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:44:12.325835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:44:12.339682 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:44:12.341106 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:44:12.352026 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1259/2529] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:44:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:44:13.344581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978586b-11b9-47d0-8d3d-333d6e1af755 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:44:13.406062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978586b-11b9-47d0-8d3d-333d6e1af755 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:44:13.410355 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:44:13.410610 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:44:13.410890 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:44:13.411170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:44:13.413008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:44:13.427242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:44:13.428851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:44:13.442154 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1271/2530] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:44:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:44:16.592525 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:44:16.602254 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e4aa30-d8c6-4745-ad7b-f0bee9b5c8ca service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:44:16.605856 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:44:16.606463 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:44:16.606753 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:44:16.607147 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:44:16.608868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:44:16.619057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:44:16.620559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:44:16.628907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1260/2531] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:44:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:44:23.438421 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=26 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:44:26.625009 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=32 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:44:31.368065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:44:31.376863 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa9c1c9-b61d-420d-b85b-256de91b3504 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:44:31.381271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:44:31.381928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:44:31.382239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:44:31.382544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:44:31.391919 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:44:31.393362 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:44:31.403658 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1272/2532] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:44:31 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:44:41.399394 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:03.151152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:03.152680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:03.271553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8a1f9f-58f3-467e-a69e-dc8b219446e9 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:03.277543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:03.278162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:03.278446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:03.278724 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:03.280440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:03.291854 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:03.293385 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:03.303155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1261/2533] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:13.298961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=54 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:30.765245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:30.774555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d415069-a1a8-4e63-b70c-f550820adf6e service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:30.778453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:30.779092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:30.779370 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:30.779639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:30.781387 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:30.843584 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:30.845002 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:30.855591 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1273/2534] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:34.329893 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:34.331344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:34.344204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45bf0f9-567b-45ac-bad2-5cc5be83c3e5 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:34.349315 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:34.349952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:34.350236 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:34.350511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:34.352244 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:34.364855 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:34.366483 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:34.377203 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1262/2535] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:34.443187 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:34.444684 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:34.458247 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28014fa5-57f9-4441-ba4b-1dec92141302 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:34.464129 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:34.464773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:34.465062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:34.465373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] PATH_INFO: `/v3/users/724baaa4aa834e609fe4f513f9fc0b4b` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:34.466513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=724baaa4aa834e609fe4f513f9fc0b4b)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:34.468456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:34.644481 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '724baaa4aa834e609fe4f513f9fc0b4b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:34.645057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '724baaa4aa834e609fe4f513f9fc0b4b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:34.657057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '724baaa4aa834e609fe4f513f9fc0b4b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:34.663924 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '724baaa4aa834e609fe4f513f9fc0b4b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:34.668618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-bc45423f-88a9-470b-b487-e7221493bef4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '724baaa4aa834e609fe4f513f9fc0b4b'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:34.751422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1274/2536] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:45:34 2022] DELETE /identity/v3/users/724baaa4aa834e609fe4f513f9fc0b4b => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:34.974607 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:34.988547 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb85258-ba99-4be6-b33f-9e5511a157e0 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:35.047972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:35.048634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.048919 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.049235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] PATH_INFO: `/v3/projects/940ce72c42534c5196dfb909cc2a8d54` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.050383 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=940ce72c42534c5196dfb909cc2a8d54)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:35.052306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:35.150448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '940ce72c42534c5196dfb909cc2a8d54', 'request_id': 'req-af5e59c5-ffed-4127-96e7-46262967af43'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:35.236209 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '940ce72c42534c5196dfb909cc2a8d54', 'request_id': 'req-af5e59c5-ffed-4127-96e7-46262967af43'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:35.242924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '940ce72c42534c5196dfb909cc2a8d54', 'request_id': 'req-af5e59c5-ffed-4127-96e7-46262967af43'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:35.254235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] The token cache is being invalidate because project 940ce72c42534c5196dfb909cc2a8d54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:45:35.256791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:35.256791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-af5e59c5-ffed-4127-96e7-46262967af43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:35.256791 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1263/2537] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:45:34 2022] DELETE /identity/v3/projects/940ce72c42534c5196dfb909cc2a8d54 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:35.263359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac23f39-70c1-4810-b827-aa6f60605f56 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.263689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac23f39-70c1-4810-b827-aa6f60605f56 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.263981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac23f39-70c1-4810-b827-aa6f60605f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.283841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-5ac23f39-70c1-4810-b827-aa6f60605f56 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:35.285045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-5ac23f39-70c1-4810-b827-aa6f60605f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:35.344724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1275/2538] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:35.352659 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41dae4-60ea-4a97-85ba-6ee503ef6e80 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.353388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41dae4-60ea-4a97-85ba-6ee503ef6e80 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.353388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41dae4-60ea-4a97-85ba-6ee503ef6e80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.374749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3a41dae4-60ea-4a97-85ba-6ee503ef6e80 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:35.375949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3a41dae4-60ea-4a97-85ba-6ee503ef6e80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:35.384501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1264/2539] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:35.388656 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac23f39-70c1-4810-b827-aa6f60605f56 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:35.398343 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac23f39-70c1-4810-b827-aa6f60605f56 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:35.403288 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:35.403892 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.404173 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.404447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.405119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:45:35.405468 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:35.407098 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:35.408625 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:35.419133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1276/2540] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:45:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:45:35.425834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41dae4-60ea-4a97-85ba-6ee503ef6e80 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:35.436181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41dae4-60ea-4a97-85ba-6ee503ef6e80 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:35.440931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:35.441578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.441862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.442139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.442778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:35.444672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:35.540157 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1265/2541] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:45:35 2022] POST /identity/v3/projects => generated 365 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:35.544324 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:35.554664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb433339-7efc-491e-aab9-fb2f5dabc450 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:35.612335 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:35.612940 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.613246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.613535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.614161 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:35.616107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:35.721600 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1277/2542] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:45:35 2022] POST /identity/v3/users => generated 283 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:35.725636 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:35.737172 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3031c8-64c7-406a-b11b-61e600db3af4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:35.741881 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:35.742620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.742907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.743742 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.744536 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:35.746204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:35.757073 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1266/2543] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:35.761305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:35.771396 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6554768-2671-47fc-8192-e583b72b7f9a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:35.776092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00bf584-5287-4693-90db-9c046e74e4a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:35.776688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00bf584-5287-4693-90db-9c046e74e4a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.776971 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00bf584-5287-4693-90db-9c046e74e4a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.777250 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00bf584-5287-4693-90db-9c046e74e4a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.777920 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00bf584-5287-4693-90db-9c046e74e4a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:35.779544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00bf584-5287-4693-90db-9c046e74e4a6 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:35.790988 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1278/2544] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:35.794455 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:35.804615 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9e685b-ed5a-43ac-aff1-4bacb4b8d8c4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:35.810043 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:35.810694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.810980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.811259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] PATH_INFO: `/v3/projects/0f00ef52cdb64d6ca78f4a7f5e5e1341/users/c96097fb4c1a4dbe8b118179f006263e/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.825790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f00ef52cdb64d6ca78f4a7f5e5e1341, user_id=c96097fb4c1a4dbe8b118179f006263e, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:35.828442 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:35.916378 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1267/2545] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:45:35 2022] PUT /identity/v3/projects/0f00ef52cdb64d6ca78f4a7f5e5e1341/users/c96097fb4c1a4dbe8b118179f006263e/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:35.926329 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13ae0bf-db0f-417d-b2e7-8aeb45c50052 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:35.926617 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13ae0bf-db0f-417d-b2e7-8aeb45c50052 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:35.931278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13ae0bf-db0f-417d-b2e7-8aeb45c50052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:35.956083 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c13ae0bf-db0f-417d-b2e7-8aeb45c50052 None None] MFA Rules not processed for user `c96097fb4c1a4dbe8b118179f006263e`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:35.957279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c13ae0bf-db0f-417d-b2e7-8aeb45c50052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:36.021745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1279/2546] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:35 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:45:36.027027 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:36.040301 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:36.087699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c89fae-f497-4461-a35b-293dcd91c2c1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:36.092789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:36.093420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.093705 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.093979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.094606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:36.096495 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:36.220610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1268/2547] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:45:36 2022] POST /identity/v3/projects => generated 363 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:36.224708 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13ae0bf-db0f-417d-b2e7-8aeb45c50052 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:36.236762 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13ae0bf-db0f-417d-b2e7-8aeb45c50052 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:36.293373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:36.293985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.294270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.294545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.295178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:36.297104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:36.305985 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:36.396534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1280/2548] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:45:36 2022] POST /identity/v3/users => generated 288 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:36.400430 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:36.411508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f98571a-a04f-44a5-b53e-818f468b0e82 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:36.416204 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:36.417036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.417357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.417633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.418414 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:36.420014 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:36.430485 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1269/2549] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:36.434827 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:36.445018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183373b2-164a-42da-87fa-5868cd3f9609 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:36.450549 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:36.451270 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.451560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.451842 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] PATH_INFO: `/v3/projects/ecae20f7483543219910d8155883edb4/users/2da4ec704d3449e28114cdc61101dcaa/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.467771 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ecae20f7483543219910d8155883edb4, user_id=2da4ec704d3449e28114cdc61101dcaa, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:36.470535 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:36.538503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1281/2550] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:45:36 2022] PUT /identity/v3/projects/ecae20f7483543219910d8155883edb4/users/2da4ec704d3449e28114cdc61101dcaa/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:45:36.542428 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:36.552423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c570c8-92e2-4476-8cac-42c70b1d97be admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:36.608730 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c5a642-6970-4fc7-84b2-fda28c1c2088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:36.609358 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5a642-6970-4fc7-84b2-fda28c1c2088 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.609645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5a642-6970-4fc7-84b2-fda28c1c2088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.609922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5a642-6970-4fc7-84b2-fda28c1c2088 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.610810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c5a642-6970-4fc7-84b2-fda28c1c2088 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:36.612389 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c5a642-6970-4fc7-84b2-fda28c1c2088 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:36.622143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1270/2551] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:36 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:36.626249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:36.636692 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18638c2f-71f0-4e90-abe9-2dc81634abbb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:36.641881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:36.642534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.642817 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.643092 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] PATH_INFO: `/v3/projects/ecae20f7483543219910d8155883edb4/users/2da4ec704d3449e28114cdc61101dcaa/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.645056 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ecae20f7483543219910d8155883edb4, user_id=2da4ec704d3449e28114cdc61101dcaa, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:36.647803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:36.658698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1282/2552] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:45:36 2022] PUT /identity/v3/projects/ecae20f7483543219910d8155883edb4/users/2da4ec704d3449e28114cdc61101dcaa/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:36.670129 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b80ecf-27ca-4f34-850a-4acae9deedb8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.670426 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b80ecf-27ca-4f34-850a-4acae9deedb8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.670707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b80ecf-27ca-4f34-850a-4acae9deedb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.698825 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-48b80ecf-27ca-4f34-850a-4acae9deedb8 None None] MFA Rules not processed for user `2da4ec704d3449e28114cdc61101dcaa`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:36.700623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-48b80ecf-27ca-4f34-850a-4acae9deedb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:36.767783 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1271/2553] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:36 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:36.779258 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:36.780962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:36.839143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3e631e-2926-45f1-98e2-6f83f157e90f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:36.842782 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77cdd4f-234d-444c-95eb-23db0f265c45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:36.843383 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77cdd4f-234d-444c-95eb-23db0f265c45 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:36.843660 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77cdd4f-234d-444c-95eb-23db0f265c45 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:36.843928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77cdd4f-234d-444c-95eb-23db0f265c45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:36.852571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77cdd4f-234d-444c-95eb-23db0f265c45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:36.854027 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77cdd4f-234d-444c-95eb-23db0f265c45 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:36.863582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1283/2554] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:36 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:37.709959 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b80ecf-27ca-4f34-850a-4acae9deedb8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:37.714796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:37.766747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b80ecf-27ca-4f34-850a-4acae9deedb8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:37.771214 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:37.771836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:37.772120 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:37.772391 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] PATH_INFO: `/v3/users/c96097fb4c1a4dbe8b118179f006263e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:37.773532 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c96097fb4c1a4dbe8b118179f006263e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:37.775427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:37.912358 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c96097fb4c1a4dbe8b118179f006263e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:37.924322 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c96097fb4c1a4dbe8b118179f006263e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:37.984098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c96097fb4c1a4dbe8b118179f006263e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:37.984669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c96097fb4c1a4dbe8b118179f006263e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:37.985562 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c96097fb4c1a4dbe8b118179f006263e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:37.998488 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1272/2555] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:45:37 2022] DELETE /identity/v3/users/c96097fb4c1a4dbe8b118179f006263e => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:38.004090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:38.004601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:38.004702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:38.024141 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:38.025407 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:38.084607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1284/2556] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 20:45:38.094858 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:38.096277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:38.152228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87140522-bc43-4dbf-b23f-3f8270acb37f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:38.156744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:38.157377 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:38.157706 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:38.157988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:38.166934 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:38.168448 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:38.177747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1273/2557] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:38.598117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:38.601611 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:38.615135 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a6a0b-d38e-4283-b982-b03153138557 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:38.620710 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:38.621555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:38.622033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:38.622953 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] PATH_INFO: `/v3/projects/0f00ef52cdb64d6ca78f4a7f5e5e1341` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:38.624229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f00ef52cdb64d6ca78f4a7f5e5e1341)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:38.626189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:38.761435 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f00ef52cdb64d6ca78f4a7f5e5e1341', 'request_id': 'req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:38.763964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:INSERT=4,keystone:SELECT=91,keystone:DELETE=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:38.768020 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f00ef52cdb64d6ca78f4a7f5e5e1341', 'request_id': 'req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:38.774411 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f00ef52cdb64d6ca78f4a7f5e5e1341', 'request_id': 'req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:38.851946 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] The token cache is being invalidate because project 0f00ef52cdb64d6ca78f4a7f5e5e1341 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:45:38.852175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:38.852856 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:38.854462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1285/2558] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:45:38 2022] DELETE /identity/v3/projects/0f00ef52cdb64d6ca78f4a7f5e5e1341 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:38.858574 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:38.868489 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9628f68c-2c6f-4188-b294-893999e8472b service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:38.897017 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:38.926593 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:38.927618 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:38.927988 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:38.928267 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] PATH_INFO: `/v3/users/2da4ec704d3449e28114cdc61101dcaa` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:38.929445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=2da4ec704d3449e28114cdc61101dcaa)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:38.931394 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:39.118827 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2da4ec704d3449e28114cdc61101dcaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.130889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2da4ec704d3449e28114cdc61101dcaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.223922 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2da4ec704d3449e28114cdc61101dcaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.224553 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2da4ec704d3449e28114cdc61101dcaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.225141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c0031a6-779e-4933-9f12-e3e6469915ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2da4ec704d3449e28114cdc61101dcaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.238577 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1274/2559] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:45:38 2022] DELETE /identity/v3/users/2da4ec704d3449e28114cdc61101dcaa => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:39.660688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:39.663484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:39.677580 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1de5ad-737e-4a68-a1ba-42decc0d4fa5 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:39.736976 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:39.737695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:39.737974 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:39.738251 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] PATH_INFO: `/v3/projects/ecae20f7483543219910d8155883edb4` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:39.739368 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ecae20f7483543219910d8155883edb4)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:39.741282 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:39.853742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ecae20f7483543219910d8155883edb4', 'request_id': 'req-c98fa48f-aa6d-452a-aa1e-38c9264af144'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.860442 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ecae20f7483543219910d8155883edb4', 'request_id': 'req-c98fa48f-aa6d-452a-aa1e-38c9264af144'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.866885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ecae20f7483543219910d8155883edb4', 'request_id': 'req-c98fa48f-aa6d-452a-aa1e-38c9264af144'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.955313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] The token cache is being invalidate because project ecae20f7483543219910d8155883edb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:45:39.955700 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.956361 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-c98fa48f-aa6d-452a-aa1e-38c9264af144 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:39.957896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1286/2560] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:45:39 2022] DELETE /identity/v3/projects/ecae20f7483543219910d8155883edb4 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:39.963752 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd016b6-6a5e-4b99-9233-62d0b7b7f607 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:39.964257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd016b6-6a5e-4b99-9233-62d0b7b7f607 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:39.964354 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd016b6-6a5e-4b99-9233-62d0b7b7f607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:39.986201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-cbd016b6-6a5e-4b99-9233-62d0b7b7f607 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:39.987406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cbd016b6-6a5e-4b99-9233-62d0b7b7f607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:40.049397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1275/2561] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:40.056996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a559ce5-ba94-4d74-b7a9-e794d952f1f0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.057287 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a559ce5-ba94-4d74-b7a9-e794d952f1f0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.057601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a559ce5-ba94-4d74-b7a9-e794d952f1f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.077582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-6a559ce5-ba94-4d74-b7a9-e794d952f1f0 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:40.078913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-6a559ce5-ba94-4d74-b7a9-e794d952f1f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:40.086802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1287/2562] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:40.091645 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd016b6-6a5e-4b99-9233-62d0b7b7f607 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.101250 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd016b6-6a5e-4b99-9233-62d0b7b7f607 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.105809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.106405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.106687 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.106961 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.107600 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:45:40.107904 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.109529 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:40.121158 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1276/2563] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:45:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:45:40.125298 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a559ce5-ba94-4d74-b7a9-e794d952f1f0 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.134453 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a559ce5-ba94-4d74-b7a9-e794d952f1f0 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.139791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.140392 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.140693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.140944 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.141609 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.143513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:40.220045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1288/2564] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:45:40 2022] POST /identity/v3/projects => generated 381 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:40.224210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.234183 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45195af3-5b99-478d-9dc0-42abebc34d54 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.268503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:40.291971 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.292585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.292906 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.293684 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.294318 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.296233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:40.413571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1277/2565] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:45:40 2022] POST /identity/v3/users => generated 291 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:40.417647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.428096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a6740-0b59-47be-92cf-2e2e82061690 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.432726 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.433901 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.434291 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.439156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.439156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.439156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:40.446524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1289/2566] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:40.451032 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.460964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6574c0df-94e4-4f36-bab0-b8e82c24f60d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.465771 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c582bf48-0d4e-4783-b5d4-50b8345ee44a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.466367 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582bf48-0d4e-4783-b5d4-50b8345ee44a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.471567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582bf48-0d4e-4783-b5d4-50b8345ee44a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.471567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582bf48-0d4e-4783-b5d4-50b8345ee44a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.471567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582bf48-0d4e-4783-b5d4-50b8345ee44a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.471567 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582bf48-0d4e-4783-b5d4-50b8345ee44a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:40.479641 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1278/2567] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:40.483794 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.494233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f978a2-df23-400a-a5d5-c84f81975f0d admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.499384 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.500061 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.500345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.500623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] PATH_INFO: `/v3/projects/d9b3d0668c244bd6bbbb1ec0378d051f/users/ca491543b3784aa394b12069d90a1cfe/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.515698 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9b3d0668c244bd6bbbb1ec0378d051f, user_id=ca491543b3784aa394b12069d90a1cfe, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.518399 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:40.601160 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1290/2568] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:45:40 2022] PUT /identity/v3/projects/d9b3d0668c244bd6bbbb1ec0378d051f/users/ca491543b3784aa394b12069d90a1cfe/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:40.611517 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190694d9-88b3-4c3f-9285-f27411a4cb1c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.620065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190694d9-88b3-4c3f-9285-f27411a4cb1c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.620065 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190694d9-88b3-4c3f-9285-f27411a4cb1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.640568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-190694d9-88b3-4c3f-9285-f27411a4cb1c None None] MFA Rules not processed for user `ca491543b3784aa394b12069d90a1cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:40.641819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-190694d9-88b3-4c3f-9285-f27411a4cb1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:40.706087 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1279/2569] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:40 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:40.710542 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.720045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:40.775189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697914e-daf6-4368-8890-9caf6bbe0365 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.780119 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.780732 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.781016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.781297 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.781964 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.783913 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:40.865237 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1291/2570] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:45:40 2022] POST /identity/v3/projects => generated 381 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:40.869464 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190694d9-88b3-4c3f-9285-f27411a4cb1c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:40.880808 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190694d9-88b3-4c3f-9285-f27411a4cb1c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:40.937198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:40.937838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:40.938121 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:40.938396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:40.939018 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:40.940908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:41.030729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1280/2571] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:45:40 2022] POST /identity/v3/users => generated 297 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:45:41.034559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:41.044894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1e45ea-bef9-46a4-8d86-805e4e66475f admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:41.049736 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:41.050327 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:41.050607 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:41.050880 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:41.051544 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:41.053127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:41.063885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1292/2572] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:41.068067 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:41.078928 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6615dc65-7074-4f58-a6dd-4b4e899fed5c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:41.083478 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:41.084128 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:41.084406 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:41.084678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] PATH_INFO: `/v3/projects/608c8bd64c5f48308fba36775c31c819/users/68f27f341e3840c8909b737ee2579ed2/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:41.099810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=608c8bd64c5f48308fba36775c31c819, user_id=68f27f341e3840c8909b737ee2579ed2, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:41.102480 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:41.166611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1281/2573] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:45:41 2022] PUT /identity/v3/projects/608c8bd64c5f48308fba36775c31c819/users/68f27f341e3840c8909b737ee2579ed2/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:45:41.170825 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:41.180876 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd6e7b-b182-4104-a4c7-5bb49fa1a825 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:41.237261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0580f68-6916-4c77-a0d8-ac725da63e25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:41.237884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0580f68-6916-4c77-a0d8-ac725da63e25 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:41.238163 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0580f68-6916-4c77-a0d8-ac725da63e25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:41.238436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0580f68-6916-4c77-a0d8-ac725da63e25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:41.239062 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0580f68-6916-4c77-a0d8-ac725da63e25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:41.240639 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0580f68-6916-4c77-a0d8-ac725da63e25 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:41.250894 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1293/2574] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:41 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:45:41.254746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:41.265410 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d78f47e-a86c-4e16-9f1b-1715f60e7d3c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:41.269982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:41.270629 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:41.270905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:41.271176 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] PATH_INFO: `/v3/projects/608c8bd64c5f48308fba36775c31c819/users/68f27f341e3840c8909b737ee2579ed2/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:41.273039 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] RBAC: Authorizing `identity:create_grant(project_id=608c8bd64c5f48308fba36775c31c819, user_id=68f27f341e3840c8909b737ee2579ed2, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:41.275705 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:41.281151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:41.286200 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1282/2575] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:45:41 2022] PUT /identity/v3/projects/608c8bd64c5f48308fba36775c31c819/users/68f27f341e3840c8909b737ee2579ed2/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:41.303059 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7943e6d-12b8-468d-89d1-39eee7b73776 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:41.303345 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7943e6d-12b8-468d-89d1-39eee7b73776 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:41.303626 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7943e6d-12b8-468d-89d1-39eee7b73776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:41.333080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c7943e6d-12b8-468d-89d1-39eee7b73776 None None] MFA Rules not processed for user `68f27f341e3840c8909b737ee2579ed2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:41.334326 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c7943e6d-12b8-468d-89d1-39eee7b73776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:41.404071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1294/2576] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:41 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:41.416185 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:41.418617 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:41.476259 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5a195a-d7c0-4fe2-b296-0c9ad0a6125a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:41.480025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:41.481278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:41.481592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:41.481867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:41.491069 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:41.492493 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:41.501921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1283/2577] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:41 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:42.317693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7943e6d-12b8-468d-89d1-39eee7b73776 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:42.322088 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:42.328226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7943e6d-12b8-468d-89d1-39eee7b73776 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:42.333045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68f27f341e3840c8909b737ee2579ed2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '608c8bd64c5f48308fba36775c31c819', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:42.333704 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:42.333999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:42.334279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:42.335070 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:45:42.335375 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:42.337003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:42.349007 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1295/2578] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:45:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:45:42.353162 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:42.364788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4e1ae-9c38-4595-a6bb-aa56b8b0dad3 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:42.373901 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68f27f341e3840c8909b737ee2579ed2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '608c8bd64c5f48308fba36775c31c819', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:42.374519 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:42.374807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:42.375117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:42.375799 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:42.377797 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:42.493242 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1284/2579] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:45:42 2022] POST /identity/v3/projects => generated 352 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:42.498018 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:42.507875 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d92553-94aa-4d1f-9916-5c6b8d202c08 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:42.564425 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68f27f341e3840c8909b737ee2579ed2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '608c8bd64c5f48308fba36775c31c819', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:42.564935 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:42.565225 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:42.565537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:42.566179 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:45:42.566489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:42.568087 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:42.579623 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1296/2580] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:45:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:45:42.618429 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:42.629168 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01252af3-0156-4d99-8044-88c318580854 tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:42.633754 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68f27f341e3840c8909b737ee2579ed2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '608c8bd64c5f48308fba36775c31c819', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:42.634484 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:42.634769 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:42.635068 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] PATH_INFO: `/v3/projects/3e67d1fb2dda41c48ad7f26c55632831` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:42.636181 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorizing `identity:delete_project(project_id=3e67d1fb2dda41c48ad7f26c55632831)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:42.638177 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:42.761460 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e67d1fb2dda41c48ad7f26c55632831', 'request_id': 'req-dada4d2a-dc13-41da-9fe2-41db1236ca4e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:42.829729 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e67d1fb2dda41c48ad7f26c55632831', 'request_id': 'req-dada4d2a-dc13-41da-9fe2-41db1236ca4e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:42.837987 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e67d1fb2dda41c48ad7f26c55632831', 'request_id': 'req-dada4d2a-dc13-41da-9fe2-41db1236ca4e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:42.849033 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] The token cache is being invalidate because project 3e67d1fb2dda41c48ad7f26c55632831 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:45:42.849340 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:42.850024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-dada4d2a-dc13-41da-9fe2-41db1236ca4e tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:42.851501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1285/2581] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:45:42 2022] DELETE /identity/v3/projects/3e67d1fb2dda41c48ad7f26c55632831 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:43.421902 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:43.423750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:43.491195 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08055b9-f834-49c8-86eb-ee38515269ce tempest-VolumeQuotaClassesTest-1437868797 tempest-VolumeQuotaClassesTest-1437868797-project-admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:43.547304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:43.547930 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:43.548210 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:43.548484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] PATH_INFO: `/v3/users/ca491543b3784aa394b12069d90a1cfe` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:43.549631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca491543b3784aa394b12069d90a1cfe)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:43.551539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:43.739618 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:43.752293 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca491543b3784aa394b12069d90a1cfe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:43.752959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca491543b3784aa394b12069d90a1cfe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:43.765523 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca491543b3784aa394b12069d90a1cfe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:43.772413 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca491543b3784aa394b12069d90a1cfe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:43.773057 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca491543b3784aa394b12069d90a1cfe'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:43.829773 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1297/2582] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:45:43 2022] DELETE /identity/v3/users/ca491543b3784aa394b12069d90a1cfe => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:43.834201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:43.834689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:43.834798 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:43.853545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:43.854741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:43.913933 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1286/2583] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:43.924279 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:43.925806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:43.983507 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f78f2-9809-4317-94b3-127d81bc3d3e admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:43.987512 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:43.988546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:43.988546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:43.988723 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:43.998444 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:43.999795 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:44.009149 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1298/2584] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:44.454096 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:44.455882 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:44.463720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:44.469925 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a7a5df-4e17-4256-9f31-1437aa273861 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:44.474745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:44.475422 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:44.475702 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:44.476006 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] PATH_INFO: `/v3/projects/d9b3d0668c244bd6bbbb1ec0378d051f` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:44.477834 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9b3d0668c244bd6bbbb1ec0378d051f)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:44.479852 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:44.578937 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9b3d0668c244bd6bbbb1ec0378d051f', 'request_id': 'req-20d22571-e01f-4681-8336-bd278e429ee9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.669520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9b3d0668c244bd6bbbb1ec0378d051f', 'request_id': 'req-20d22571-e01f-4681-8336-bd278e429ee9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.676599 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9b3d0668c244bd6bbbb1ec0378d051f', 'request_id': 'req-20d22571-e01f-4681-8336-bd278e429ee9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.687717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] The token cache is being invalidate because project d9b3d0668c244bd6bbbb1ec0378d051f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:45:44.688011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.688643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.690170 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1287/2585] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:45:44 2022] DELETE /identity/v3/projects/d9b3d0668c244bd6bbbb1ec0378d051f => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:44.694185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:44.704451 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582a8c79-ceda-4e3e-a91f-1390a3b16f78 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:44.762304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:44.762961 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:44.763265 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:44.763537 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] PATH_INFO: `/v3/users/68f27f341e3840c8909b737ee2579ed2` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:44.764711 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] RBAC: Authorizing `identity:delete_user(user_id=68f27f341e3840c8909b737ee2579ed2)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:44.766682 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:44.882951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68f27f341e3840c8909b737ee2579ed2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.883546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68f27f341e3840c8909b737ee2579ed2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.889354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:44.896117 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68f27f341e3840c8909b737ee2579ed2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.903508 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68f27f341e3840c8909b737ee2579ed2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.904156 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-6a71a58e-802b-4bcc-ba35-d433e7e0f339 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68f27f341e3840c8909b737ee2579ed2'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:44.955603 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1299/2586] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:45:44 2022] DELETE /identity/v3/users/68f27f341e3840c8909b737ee2579ed2 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:45.385284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:45.386935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:45.400420 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d22571-e01f-4681-8336-bd278e429ee9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:45.458537 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:45.459201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:45.459470 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:45.459749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] PATH_INFO: `/v3/projects/608c8bd64c5f48308fba36775c31c819` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:45.460860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=608c8bd64c5f48308fba36775c31c819)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:45.462819 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:45.550338 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '608c8bd64c5f48308fba36775c31c819', 'request_id': 'req-a9c654d6-e63e-48e4-87a3-5091a11ebe55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:45.600746 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '608c8bd64c5f48308fba36775c31c819', 'request_id': 'req-a9c654d6-e63e-48e4-87a3-5091a11ebe55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:45.608240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '608c8bd64c5f48308fba36775c31c819', 'request_id': 'req-a9c654d6-e63e-48e4-87a3-5091a11ebe55'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:45.618903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] The token cache is being invalidate because project 608c8bd64c5f48308fba36775c31c819 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:45:45.619198 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:45.619835 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a9c654d6-e63e-48e4-87a3-5091a11ebe55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:45:45.622472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1288/2587] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:45:45 2022] DELETE /identity/v3/projects/608c8bd64c5f48308fba36775c31c819 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:45.627993 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94a3f2-1f3d-451f-8202-b4487fa75d79 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:45.628328 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94a3f2-1f3d-451f-8202-b4487fa75d79 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:45.628568 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94a3f2-1f3d-451f-8202-b4487fa75d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:45.648776 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-ea94a3f2-1f3d-451f-8202-b4487fa75d79 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:45.650077 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ea94a3f2-1f3d-451f-8202-b4487fa75d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:45.710207 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1300/2588] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:45.719382 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30de9fb-e5a9-4433-afda-e16ba93c095e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:45.719674 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30de9fb-e5a9-4433-afda-e16ba93c095e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:45.719980 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30de9fb-e5a9-4433-afda-e16ba93c095e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:45.739788 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f30de9fb-e5a9-4433-afda-e16ba93c095e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:45.740992 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f30de9fb-e5a9-4433-afda-e16ba93c095e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:45.749520 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1289/2589] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:45.753683 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea94a3f2-1f3d-451f-8202-b4487fa75d79 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:45.763965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea94a3f2-1f3d-451f-8202-b4487fa75d79 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:45.768492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:45.769080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:45.769393 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:45.769667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:45.770304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:45:45.770605 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:45.772191 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:45.783446 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1301/2590] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:45:45.787681 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30de9fb-e5a9-4433-afda-e16ba93c095e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:45.797623 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30de9fb-e5a9-4433-afda-e16ba93c095e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:45.802405 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:45.803007 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:45.803320 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:45.803592 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:45.804218 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:45.806190 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:45.812079 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:45.933945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1290/2591] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:45:45 2022] POST /identity/v3/projects => generated 369 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:45:45.937593 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:45.947645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee38bbe-db1a-471a-81d3-1c1495f47490 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:46.004261 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:46.004791 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:46.005076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:46.005382 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:46.006021 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:46.007998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:46.081102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1302/2592] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:45:45 2022] POST /identity/v3/users => generated 285 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:45:46.085238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:46.095672 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a26840c-1185-407f-8064-6e0f604780db admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:46.100380 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:46.100984 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:46.101270 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:46.101585 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:46.102227 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:46.103850 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:46.115070 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1291/2593] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:45:46.118885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:46.129785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998fafcd-dfcc-493c-b5d7-2197a5355118 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:46.134585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a759b629-7622-4c9b-a229-d7202b5407de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:46.135218 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a759b629-7622-4c9b-a229-d7202b5407de admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:46.135500 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a759b629-7622-4c9b-a229-d7202b5407de admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:46.135777 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a759b629-7622-4c9b-a229-d7202b5407de admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:46.136439 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a759b629-7622-4c9b-a229-d7202b5407de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:46.138956 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a759b629-7622-4c9b-a229-d7202b5407de admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:46.150851 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1303/2594] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:45:46.154349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:46.164346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ffa0e2-41a5-4a1c-9d10-8acb02c99677 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:46.169589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:46.170251 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:46.170653 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:46.170936 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] PATH_INFO: `/v3/projects/1ce5d1c1f0ab48999c43202cacc99ced/users/55e098d11bc44cf68667d23c457438ee/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:46.186329 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ce5d1c1f0ab48999c43202cacc99ced, user_id=55e098d11bc44cf68667d23c457438ee, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:46.188979 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:46.273634 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1292/2595] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:45:46 2022] PUT /identity/v3/projects/1ce5d1c1f0ab48999c43202cacc99ced/users/55e098d11bc44cf68667d23c457438ee/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:45:46.317550 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782d8e4-05b4-4a3e-ab8c-bc86f78ecfff None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:46.317837 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782d8e4-05b4-4a3e-ab8c-bc86f78ecfff None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:46.318111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782d8e4-05b4-4a3e-ab8c-bc86f78ecfff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:46.345490 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-c782d8e4-05b4-4a3e-ab8c-bc86f78ecfff None None] MFA Rules not processed for user `55e098d11bc44cf68667d23c457438ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:45:46.346681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c782d8e4-05b4-4a3e-ab8c-bc86f78ecfff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:46.379800 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:46.422104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1304/2596] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:45:46 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:45:46.434836 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:45:46.436649 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:45:46.495151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01fe97a-af25-4cdd-a999-eb3586cbd339 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:45:46.499311 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977a395-4f38-487c-a2f5-41cdecf5aec8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7fae443ad92439aadd2f94ad3bca594', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:45:46.499920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977a395-4f38-487c-a2f5-41cdecf5aec8 service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:45:46.500212 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977a395-4f38-487c-a2f5-41cdecf5aec8 service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:45:46.500492 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977a395-4f38-487c-a2f5-41cdecf5aec8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:45:46.508810 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977a395-4f38-487c-a2f5-41cdecf5aec8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:45:46.510284 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977a395-4f38-487c-a2f5-41cdecf5aec8 service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:45:46.519589 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1293/2597] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:45:46 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:45:56.509095 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=15 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:45:56.515575 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=46,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:46:21.752294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c782d8e4-05b4-4a3e-ab8c-bc86f78ecfff None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:21.819695 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c782d8e4-05b4-4a3e-ab8c-bc86f78ecfff None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:21.824410 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:21.825069 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:21.825374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:21.825652 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] PATH_INFO: `/v3/users/55e098d11bc44cf68667d23c457438ee` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:21.826943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55e098d11bc44cf68667d23c457438ee)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:21.829534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:21.973045 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55e098d11bc44cf68667d23c457438ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:21.973763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55e098d11bc44cf68667d23c457438ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:21.987447 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55e098d11bc44cf68667d23c457438ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:21.995076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55e098d11bc44cf68667d23c457438ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:21.995715 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55e098d11bc44cf68667d23c457438ee'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:22.050132 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1305/2598] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:46:21 2022] DELETE /identity/v3/users/55e098d11bc44cf68667d23c457438ee => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:46:22.055210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:22.055755 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:22.055862 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:22.077341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:46:22.078559 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:22.138820 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1294/2599] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:46:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:46:22.149133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:22.150646 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:22.233313 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3744b471-eb4d-494f-8920-5a6690a5a8a1 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:22.237342 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf573a8-d9f9-4ae3-b63c-2c7ef0387e5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:22.238006 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf573a8-d9f9-4ae3-b63c-2c7ef0387e5e service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:22.238236 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf573a8-d9f9-4ae3-b63c-2c7ef0387e5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:22.238513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf573a8-d9f9-4ae3-b63c-2c7ef0387e5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:22.244484 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:46:22.248063 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf573a8-d9f9-4ae3-b63c-2c7ef0387e5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:22.249514 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf573a8-d9f9-4ae3-b63c-2c7ef0387e5e service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:22.259102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1306/2600] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:46:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:46:22.674896 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:22.680690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:22.697005 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d4ed83-9e62-4882-b663-1d7877371109 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:22.702781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:22.703427 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:22.703616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:22.703900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] PATH_INFO: `/v3/projects/1ce5d1c1f0ab48999c43202cacc99ced` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:22.705186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ce5d1c1f0ab48999c43202cacc99ced)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:22.707278 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:22.844201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ce5d1c1f0ab48999c43202cacc99ced', 'request_id': 'req-c8b6ff1f-1084-4934-ac72-b396f16a6a34'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:22.918139 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ce5d1c1f0ab48999c43202cacc99ced', 'request_id': 'req-c8b6ff1f-1084-4934-ac72-b396f16a6a34'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:22.924500 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ce5d1c1f0ab48999c43202cacc99ced', 'request_id': 'req-c8b6ff1f-1084-4934-ac72-b396f16a6a34'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:22.934809 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] The token cache is being invalidate because project 1ce5d1c1f0ab48999c43202cacc99ced was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:46:22.935094 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:22.935712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c8b6ff1f-1084-4934-ac72-b396f16a6a34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:46:22.937167 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1295/2601] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:46:22 2022] DELETE /identity/v3/projects/1ce5d1c1f0ab48999c43202cacc99ced => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:46:22.942162 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220d67ad-2bcc-4e6f-a4f5-a75cdec9b127 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:22.942638 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220d67ad-2bcc-4e6f-a4f5-a75cdec9b127 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:22.942741 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220d67ad-2bcc-4e6f-a4f5-a75cdec9b127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:22.963950 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-220d67ad-2bcc-4e6f-a4f5-a75cdec9b127 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:46:22.965168 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-220d67ad-2bcc-4e6f-a4f5-a75cdec9b127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:23.027146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1307/2602] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:46:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:46:23.035103 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956f6510-9f5c-403c-b5f8-191663d3eae2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.035400 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956f6510-9f5c-403c-b5f8-191663d3eae2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.035680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956f6510-9f5c-403c-b5f8-191663d3eae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.056708 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-956f6510-9f5c-403c-b5f8-191663d3eae2 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:46:23.057946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-956f6510-9f5c-403c-b5f8-191663d3eae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:23.065839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1296/2603] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:46:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:46:23.070262 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220d67ad-2bcc-4e6f-a4f5-a75cdec9b127 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:23.079760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220d67ad-2bcc-4e6f-a4f5-a75cdec9b127 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:23.085311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:23.085933 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.086219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.086502 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.087148 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81614) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:46:23.087454 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:23.089449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:23.101850 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1308/2604] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:46:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:46:23.105994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956f6510-9f5c-403c-b5f8-191663d3eae2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:23.115528 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956f6510-9f5c-403c-b5f8-191663d3eae2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:23.120109 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:23.120696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.120976 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.121248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.121908 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:23.123790 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:23.204837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1297/2605] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:46:23 2022] POST /identity/v3/projects => generated 361 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:46:23.210239 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:23.220602 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd83605-a9e5-4cc1-9ad7-9010df2441a5 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:23.275962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:23.276560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.276841 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.277115 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] PATH_INFO: `/v3/users` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.277806 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:23.279696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:23.358802 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1309/2606] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:46:23 2022] POST /identity/v3/users => generated 281 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:46:23.362839 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:23.373446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685f9b9d-6705-4dcb-ad38-b66dc7e7bc97 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:23.378056 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:23.378666 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.378949 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.379233 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.379868 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:23.382247 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:23.392115 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1298/2607] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:46:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:46:23.395852 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:23.406359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be64530-4016-4676-bf13-4b91863a1f44 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:23.410898 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27cdfe6-4718-4b17-b0f8-70bfc02124dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:23.411483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27cdfe6-4718-4b17-b0f8-70bfc02124dd admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.411821 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27cdfe6-4718-4b17-b0f8-70bfc02124dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.412331 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27cdfe6-4718-4b17-b0f8-70bfc02124dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.413029 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27cdfe6-4718-4b17-b0f8-70bfc02124dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:23.414650 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27cdfe6-4718-4b17-b0f8-70bfc02124dd admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:23.424957 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1310/2608] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:46:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:46:23.429228 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:23.439307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df813b3a-b1da-47a3-8b5c-64bbf6d32d3f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:23.445003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:23.445707 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.446024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.446297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] PATH_INFO: `/v3/projects/3f236f9ac07a46609d552a1606f1843c/users/9ef582eb5eff40db906e28f5cde29483/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.461734 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f236f9ac07a46609d552a1606f1843c, user_id=9ef582eb5eff40db906e28f5cde29483, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:23.465205 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:23.564257 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1299/2609] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:46:23 2022] PUT /identity/v3/projects/3f236f9ac07a46609d552a1606f1843c/users/9ef582eb5eff40db906e28f5cde29483/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:46:23.574785 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.575073 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.575350 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.604159 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] MFA Rules not processed for user `9ef582eb5eff40db906e28f5cde29483`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:46:23.605521 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:23.653364 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:46:23.670962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1311/2610] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:46:23 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:46:23.682616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:23.684439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:23.706720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:46:23.742024 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a1e78-b301-4229-a8af-3e409789ba87 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:23.745747 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:23.746568 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:23.747118 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:23.747815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:23.756773 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:23.758249 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:23.766900 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1300/2611] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:46:23 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:46:33.718601 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=5 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:46:33.804363 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=17 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:46:42.533551 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:42.535196 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:42.595560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37619df-4ce8-4de5-bf10-c3d3b043dac0 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:42.600560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:42.601172 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:42.601486 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:42.601760 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:42.603492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:42.615689 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:42.617166 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:42.628315 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1312/2612] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:46:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:46:43.779986 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:43.782469 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:43.866319 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54014d01-644c-4ac5-8b60-1f9b1f916457 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:43.872404 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:43.872781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:43.873063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:43.873359 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:43.875996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:43.888346 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:43.890700 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:43.905508 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1301/2613] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:46:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:46:44.061304 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:44.064570 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:44.075561 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d075ed-3a0a-4667-a483-f9b4e93ae911 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:44.079962 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:44.080573 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:44.080860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:44.081142 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:44.088143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:44.092696 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:44.094830 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:44.103352 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1313/2614] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:46:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:46:45.203486 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:45.213951 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c761fefa-ee1c-4677-9a4f-9926c92d2eca service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:45.217717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:45.218325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:45.218611 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:45.218887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:45.220872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:45.234741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:45.236211 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:45.247325 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1302/2615] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:46:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:46:45.312104 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:46:45.314071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:45.385745 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28a4b6-2fc1-4719-881d-a77a9e153a10 service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:46:45.390716 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdf85e9-3151-43c0-ab41-310eac204189 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:46:45.391325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf85e9-3151-43c0-ab41-310eac204189 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:46:45.391824 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf85e9-3151-43c0-ab41-310eac204189 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:46:45.392323 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf85e9-3151-43c0-ab41-310eac204189 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:46:45.394105 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-dcdf85e9-3151-43c0-ab41-310eac204189 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:46:45.407994 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdf85e9-3151-43c0-ab41-310eac204189 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:46:45.409815 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdf85e9-3151-43c0-ab41-310eac204189 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:46:45.420515 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1314/2616] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:46:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:46:55.242305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=40 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:46:55.415412 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=64 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:03.134104 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:03.196443 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf40d82-8064-4b58-a8aa-026bff162271 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:03.200905 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:03.201551 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:03.201837 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:03.202110 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] PATH_INFO: `/v3/users/9ef582eb5eff40db906e28f5cde29483` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:03.203210 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ef582eb5eff40db906e28f5cde29483)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:03.205307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:03.359330 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ef582eb5eff40db906e28f5cde29483'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:03.372299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ef582eb5eff40db906e28f5cde29483'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:03.413034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ef582eb5eff40db906e28f5cde29483'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:03.413669 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ef582eb5eff40db906e28f5cde29483'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:03.414306 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ef582eb5eff40db906e28f5cde29483'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:03.427447 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1303/2617] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:47:03 2022] DELETE /identity/v3/users/9ef582eb5eff40db906e28f5cde29483 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:03.431869 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:03.432356 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:03.432466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:03.452829 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:03.454462 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:03.516295 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1315/2618] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:03.526587 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:03.606606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:03.606606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949ee15-b74a-4fda-8a11-7f582d2332e3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:03.606606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f8e542-faf6-4bf3-8eeb-a236f2ce132f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:03.608717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f8e542-faf6-4bf3-8eeb-a236f2ce132f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:03.608717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f8e542-faf6-4bf3-8eeb-a236f2ce132f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:03.608717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f8e542-faf6-4bf3-8eeb-a236f2ce132f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:03.608717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f8e542-faf6-4bf3-8eeb-a236f2ce132f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:03.608717 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f8e542-faf6-4bf3-8eeb-a236f2ce132f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:03.617434 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1304/2619] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:47:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:47:04.021305 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.022780 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:04.035739 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ce6487-e322-4056-b6af-840ff1c24f63 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:04.041005 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:04.041702 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.041990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.042271 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] PATH_INFO: `/v3/projects/3f236f9ac07a46609d552a1606f1843c` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.043426 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f236f9ac07a46609d552a1606f1843c)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:04.045458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:04.145640 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f236f9ac07a46609d552a1606f1843c', 'request_id': 'req-f608df4d-70cc-4af8-bcf4-869a4444c6f8'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:04.151513 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f236f9ac07a46609d552a1606f1843c', 'request_id': 'req-f608df4d-70cc-4af8-bcf4-869a4444c6f8'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:04.158033 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f236f9ac07a46609d552a1606f1843c', 'request_id': 'req-f608df4d-70cc-4af8-bcf4-869a4444c6f8'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:04.237111 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] The token cache is being invalidate because project 3f236f9ac07a46609d552a1606f1843c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:47:04.237434 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:04.238064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f608df4d-70cc-4af8-bcf4-869a4444c6f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:04.239597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1316/2620] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:47:04 2022] DELETE /identity/v3/projects/3f236f9ac07a46609d552a1606f1843c => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:04.244584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b480649-aa95-41a5-819c-3147199d1e5c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.245071 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b480649-aa95-41a5-819c-3147199d1e5c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.246920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b480649-aa95-41a5-819c-3147199d1e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.265757 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3b480649-aa95-41a5-819c-3147199d1e5c None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:04.266946 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b480649-aa95-41a5-819c-3147199d1e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:04.283597 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:04.325616 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1305/2621] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:04.333761 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f96075-6dd6-4bcb-b1e1-fac0a69e6bfc None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.334072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f96075-6dd6-4bcb-b1e1-fac0a69e6bfc None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.336367 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f96075-6dd6-4bcb-b1e1-fac0a69e6bfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.354336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-a3f96075-6dd6-4bcb-b1e1-fac0a69e6bfc None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:04.355524 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-a3f96075-6dd6-4bcb-b1e1-fac0a69e6bfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:04.364212 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1317/2622] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:04.368584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b480649-aa95-41a5-819c-3147199d1e5c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.379057 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b480649-aa95-41a5-819c-3147199d1e5c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:04.383690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:04.384511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.384972 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.385639 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.386413 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:47:04.386720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:04.388310 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:04.399978 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1306/2623] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:47:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:47:04.404317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f96075-6dd6-4bcb-b1e1-fac0a69e6bfc None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.413685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f96075-6dd6-4bcb-b1e1-fac0a69e6bfc None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:04.418422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:04.419023 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.419311 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.419588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.420219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:04.422175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:04.499923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1318/2624] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:47:04 2022] POST /identity/v3/projects => generated 373 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:47:04.504295 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.514260 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b167f0-0b80-4aa1-ba3f-63f18685a26e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:04.570194 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:04.570805 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.571093 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.571371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.572011 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:04.573977 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:04.648353 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1307/2625] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:47:04 2022] POST /identity/v3/users => generated 287 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:47:04.652664 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.666358 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a5a989-e50c-463d-b037-afb80c56bc11 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:04.671843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:04.672456 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.672730 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.672996 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.674487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:04.676091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:04.686770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1319/2626] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:47:04 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:47:04.692471 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.702796 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fac95d-3737-49d4-8c2d-857c2e1401ee admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:04.707777 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f2a42a-9bfd-4f03-a96e-4c4cec9ec3c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:04.711579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2a42a-9bfd-4f03-a96e-4c4cec9ec3c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.711579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2a42a-9bfd-4f03-a96e-4c4cec9ec3c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.711579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2a42a-9bfd-4f03-a96e-4c4cec9ec3c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.711579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f2a42a-9bfd-4f03-a96e-4c4cec9ec3c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:04.711579 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f2a42a-9bfd-4f03-a96e-4c4cec9ec3c3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:04.724034 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1308/2627] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:47:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:47:04.728080 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.739125 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc742c2-81a7-4dd2-9599-d8196f8d0e4b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:04.743778 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:04.744416 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.744706 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.744981 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] PATH_INFO: `/v3/projects/733d8890a2164ec8a5d2a010b34b9676/users/b3d3f0cc4a8145edb16cd71f56dcbf42/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.760294 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=733d8890a2164ec8a5d2a010b34b9676, user_id=b3d3f0cc4a8145edb16cd71f56dcbf42, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:04.763055 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:04.831497 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1320/2628] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:47:04 2022] PUT /identity/v3/projects/733d8890a2164ec8a5d2a010b34b9676/users/b3d3f0cc4a8145edb16cd71f56dcbf42/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:04.842201 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258dedf1-adfd-413a-aa01-597e9c526514 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:04.842496 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258dedf1-adfd-413a-aa01-597e9c526514 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:04.842781 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258dedf1-adfd-413a-aa01-597e9c526514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:04.876699 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-258dedf1-adfd-413a-aa01-597e9c526514 None None] MFA Rules not processed for user `b3d3f0cc4a8145edb16cd71f56dcbf42`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:04.878886 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-258dedf1-adfd-413a-aa01-597e9c526514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:04.950117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1309/2629] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:04 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:04.961758 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:04.963455 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:04.985769 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:05.033205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d60f87-850f-46e6-b65b-ec5e3e7db8d2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:05.037714 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef9c09b-0c8b-4033-ac97-39b73697e6c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:05.038330 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef9c09b-0c8b-4033-ac97-39b73697e6c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:05.038615 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef9c09b-0c8b-4033-ac97-39b73697e6c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:05.038891 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef9c09b-0c8b-4033-ac97-39b73697e6c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:05.047374 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef9c09b-0c8b-4033-ac97-39b73697e6c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:05.049859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef9c09b-0c8b-4033-ac97-39b73697e6c7 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:05.059188 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1321/2630] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:47:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:47:05.084749 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258dedf1-adfd-413a-aa01-597e9c526514 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:05.106511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:05.152872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258dedf1-adfd-413a-aa01-597e9c526514 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:05.157299 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:05.158098 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:05.158388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:05.158660 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] PATH_INFO: `/v3/users/b3d3f0cc4a8145edb16cd71f56dcbf42` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:05.159779 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3d3f0cc4a8145edb16cd71f56dcbf42)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:05.161722 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:05.307816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3d3f0cc4a8145edb16cd71f56dcbf42'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:05.320524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3d3f0cc4a8145edb16cd71f56dcbf42'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:05.401838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3d3f0cc4a8145edb16cd71f56dcbf42'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:05.402481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3d3f0cc4a8145edb16cd71f56dcbf42'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:05.403088 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3d3f0cc4a8145edb16cd71f56dcbf42'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:05.415738 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1310/2631] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:47:05 2022] DELETE /identity/v3/users/b3d3f0cc4a8145edb16cd71f56dcbf42 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:05.420301 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37148d8d-992c-407e-891f-e7502300504e None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:05.420643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37148d8d-992c-407e-891f-e7502300504e None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:05.420926 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37148d8d-992c-407e-891f-e7502300504e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:05.440205 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-37148d8d-992c-407e-891f-e7502300504e None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:05.441436 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-37148d8d-992c-407e-891f-e7502300504e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:05.504341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1322/2632] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:05.513889 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:05.515343 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:05.576685 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83b8efc-b354-4c34-9ee3-54261fe1c05f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:05.580985 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d30218-0286-4ba5-9d9e-01886833ba98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:05.581626 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d30218-0286-4ba5-9d9e-01886833ba98 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:05.581915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d30218-0286-4ba5-9d9e-01886833ba98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:05.582193 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d30218-0286-4ba5-9d9e-01886833ba98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:05.591095 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d30218-0286-4ba5-9d9e-01886833ba98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:05.592524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d30218-0286-4ba5-9d9e-01886833ba98 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:05.601910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1311/2633] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:47:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:47:06.038106 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37148d8d-992c-407e-891f-e7502300504e None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.039536 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-37148d8d-992c-407e-891f-e7502300504e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:06.052363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37148d8d-992c-407e-891f-e7502300504e None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:06.057025 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:06.057724 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.058037 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.058318 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] PATH_INFO: `/v3/projects/733d8890a2164ec8a5d2a010b34b9676` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.059437 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=733d8890a2164ec8a5d2a010b34b9676)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:06.061391 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:06.162763 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '733d8890a2164ec8a5d2a010b34b9676', 'request_id': 'req-f414ada8-085b-46ab-a6f7-2be2491530d8'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:06.168688 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '733d8890a2164ec8a5d2a010b34b9676', 'request_id': 'req-f414ada8-085b-46ab-a6f7-2be2491530d8'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:06.175473 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '733d8890a2164ec8a5d2a010b34b9676', 'request_id': 'req-f414ada8-085b-46ab-a6f7-2be2491530d8'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:06.258257 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] The token cache is being invalidate because project 733d8890a2164ec8a5d2a010b34b9676 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81614) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:47:06.258578 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:06.259215 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-f414ada8-085b-46ab-a6f7-2be2491530d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:47:06.260740 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1323/2634] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:47:06 2022] DELETE /identity/v3/projects/733d8890a2164ec8a5d2a010b34b9676 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:06.265665 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe25514e-3ed8-4163-86e6-47a4624f2dad None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.265956 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe25514e-3ed8-4163-86e6-47a4624f2dad None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.266238 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe25514e-3ed8-4163-86e6-47a4624f2dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.286606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fe25514e-3ed8-4163-86e6-47a4624f2dad None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:06.287745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fe25514e-3ed8-4163-86e6-47a4624f2dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:06.324342 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:06.350549 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1312/2635] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:06.359709 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0108fda-5c76-4f61-b9d9-22c97bf81c06 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.360003 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0108fda-5c76-4f61-b9d9-22c97bf81c06 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.360292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0108fda-5c76-4f61-b9d9-22c97bf81c06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.381936 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-e0108fda-5c76-4f61-b9d9-22c97bf81c06 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:06.383151 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-e0108fda-5c76-4f61-b9d9-22c97bf81c06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:06.391090 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1324/2636] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:06.396161 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe25514e-3ed8-4163-86e6-47a4624f2dad None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.405907 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe25514e-3ed8-4163-86e6-47a4624f2dad None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:06.412108 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:06.412712 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.412999 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.413277 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.413966 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 20:47:06.414275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:06.415867 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:06.428255 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1313/2637] 10.222.0.66 () {58 vars in 1153 bytes} [Fri Oct 28 20:47:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 20:47:06.432516 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0108fda-5c76-4f61-b9d9-22c97bf81c06 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.442377 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0108fda-5c76-4f61-b9d9-22c97bf81c06 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:06.446951 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:06.447597 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.447881 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.448152 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.448770 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:06.450758 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:06.534242 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1325/2638] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:47:06 2022] POST /identity/v3/projects => generated 385 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:47:06.538396 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.549741 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19c1d4f-167f-4e64-9e07-88efd00db458 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:06.605543 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:06.606142 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.606423 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.606696 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.607584 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:06.609514 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:06.705994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1314/2639] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:47:06 2022] POST /identity/v3/users => generated 293 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:47:06.710133 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.722185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c17f0d0-a16d-4cc0-b71b-b23e001481c9 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:06.726923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:06.727556 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.727843 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.728121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.728767 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:06.730555 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:06.732884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:06.742216 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1326/2640] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:47:06.748610 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.758776 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d663d4b-f3b7-4720-9c94-8b0fdcfa15c7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:06.763527 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef621290-fa2e-4e48-917d-a79df91919c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:06.768283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef621290-fa2e-4e48-917d-a79df91919c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.768283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef621290-fa2e-4e48-917d-a79df91919c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.768283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef621290-fa2e-4e48-917d-a79df91919c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.768283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef621290-fa2e-4e48-917d-a79df91919c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:06.768283 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef621290-fa2e-4e48-917d-a79df91919c0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:06.778397 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1315/2641] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:47:06.782354 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.793116 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f35a22-31b1-48e3-9ee0-8875c04d7766 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:06.797990 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:06.799177 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] REQUEST_METHOD: `PUT` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.799631 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.799910 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] PATH_INFO: `/v3/projects/7b5c9ffe3749453b8d677e173d102be1/users/8daa1cb0ade042589bc26ecaaf59b0f7/roles/f1f68385aeb8421b80010aaf19d20d6e` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.814680 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b5c9ffe3749453b8d677e173d102be1, user_id=8daa1cb0ade042589bc26ecaaf59b0f7, role_id=f1f68385aeb8421b80010aaf19d20d6e)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:06.817307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:06.887878 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1327/2642] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:47:06 2022] PUT /identity/v3/projects/7b5c9ffe3749453b8d677e173d102be1/users/8daa1cb0ade042589bc26ecaaf59b0f7/roles/f1f68385aeb8421b80010aaf19d20d6e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:06.893025 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2e3a9-4657-404b-aacc-8e2d5224d927 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:06.893313 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2e3a9-4657-404b-aacc-8e2d5224d927 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:06.893620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2e3a9-4657-404b-aacc-8e2d5224d927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:06.913366 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-87b2e3a9-4657-404b-aacc-8e2d5224d927 None None] MFA Rules not processed for user `8481404e88e44e68b2c70669768eebe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:06.914569 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-87b2e3a9-4657-404b-aacc-8e2d5224d927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:06.972051 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1316/2643] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:06.981581 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:06.983534 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:07.041430 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8871a4-6024-4c54-b9f3-8cc70ed7b956 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:07.045071 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:07.045657 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:07.045943 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:07.046219 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:07.055707 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:07.057147 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:07.066681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1328/2644] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:47:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:47:12.270390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fdb588-0fe6-4871-bcfb-145cb5abcfa9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:12.270390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fdb588-0fe6-4871-bcfb-145cb5abcfa9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:12.270390 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fdb588-0fe6-4871-bcfb-145cb5abcfa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:12.305678 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-78fdb588-0fe6-4871-bcfb-145cb5abcfa9 None None] MFA Rules not processed for user `8daa1cb0ade042589bc26ecaaf59b0f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:12.307082 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-78fdb588-0fe6-4871-bcfb-145cb5abcfa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:12.351524 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:12.387141 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1317/2645] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:12 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:12.392155 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:12.402912 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59b3f7-9144-4727-a0b5-0d838e36b9ba service neutron] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:12.409175 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:12.409955 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:12.410238 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:12.410518 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:12.411170 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:12.413086 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:12.501594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1329/2646] 10.222.0.66 () {60 vars in 1153 bytes} [Fri Oct 28 20:47:12 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 20:47:12.505633 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fdb588-0fe6-4871-bcfb-145cb5abcfa9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:12.517743 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fdb588-0fe6-4871-bcfb-145cb5abcfa9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:12.585590 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:12.586341 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:12.586688 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:12.587036 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:12.587792 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:12.590000 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:12.660300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1318/2647] 10.222.0.66 () {60 vars in 1144 bytes} [Fri Oct 28 20:47:12 2022] POST /identity/v3/users => generated 299 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 20:47:12.664325 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:12.674948 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82825e06-6f3a-4580-951f-6b806fc267e2 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:12.679742 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:12.680363 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:12.680647 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:12.680928 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:12.681616 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:12.683226 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:12.695687 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1330/2648] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:47:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 20:47:12.699857 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:12.711371 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9272b81d-a21b-4b6f-908b-a2f768793de3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:12.716240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:12.716931 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:12.717240 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:12.717613 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] PATH_INFO: `/v3/projects/dd1881d4230046a1a465f723587e6421/users/46a97178b6e34e5585b353a0bd730c23/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:12.739208 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd1881d4230046a1a465f723587e6421, user_id=46a97178b6e34e5585b353a0bd730c23, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:12.741947 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:12.842412 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1319/2649] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:47:12 2022] PUT /identity/v3/projects/dd1881d4230046a1a465f723587e6421/users/46a97178b6e34e5585b353a0bd730c23/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:12.846673 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:12.857310 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee3ae69-bdf0-443b-9963-0c4d713b48b3 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:12.914788 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2774ea-32dc-4102-9e8b-9b62bc6e5483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:12.915378 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2774ea-32dc-4102-9e8b-9b62bc6e5483 admin admin] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:12.915667 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2774ea-32dc-4102-9e8b-9b62bc6e5483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:12.915998 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2774ea-32dc-4102-9e8b-9b62bc6e5483 admin admin] PATH_INFO: `/v3/roles` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:12.916634 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2774ea-32dc-4102-9e8b-9b62bc6e5483 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:12.918278 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2774ea-32dc-4102-9e8b-9b62bc6e5483 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:12.928252 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1331/2650] 10.222.0.66 () {58 vars in 1122 bytes} [Fri Oct 28 20:47:12 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 20:47:12.932452 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:12.943709 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fb73c5-659a-4af6-b70a-11b131c67386 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:12.948846 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:12.949571 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:12.949920 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:12.950130 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] PATH_INFO: `/v3/projects/dd1881d4230046a1a465f723587e6421/users/46a97178b6e34e5585b353a0bd730c23/roles/e2631d01cdf5420a8210f6ff7e885fef` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:12.952125 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd1881d4230046a1a465f723587e6421, user_id=46a97178b6e34e5585b353a0bd730c23, role_id=e2631d01cdf5420a8210f6ff7e885fef)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:12.956151 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:12.967155 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1320/2651] 10.222.0.66 () {60 vars in 1483 bytes} [Fri Oct 28 20:47:12 2022] PUT /identity/v3/projects/dd1881d4230046a1a465f723587e6421/users/46a97178b6e34e5585b353a0bd730c23/roles/e2631d01cdf5420a8210f6ff7e885fef => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:47:18.382588 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d10e0ce-89e8-4712-ad78-03dcb3833cad None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:18.383127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d10e0ce-89e8-4712-ad78-03dcb3833cad None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:18.383127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d10e0ce-89e8-4712-ad78-03dcb3833cad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:18.416292 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-2d10e0ce-89e8-4712-ad78-03dcb3833cad None None] MFA Rules not processed for user `46a97178b6e34e5585b353a0bd730c23`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:18.417559 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2d10e0ce-89e8-4712-ad78-03dcb3833cad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:18.450546 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:18.527404 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1332/2652] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:18 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:18.542440 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:18.544838 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:18.622003 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99869424-cc41-4f1b-a3e6-444cd95b6087 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:18.627392 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a7fae443ad92439aadd2f94ad3bca594', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:18.628903 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:18.629564 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:18.629815 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:18.676831 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:18.694503 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:18.696117 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:18.706511 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1321/2653] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:47:18 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:47:19.105582 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44342626-7e60-461f-9b1d-d99c42b41415 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:19.105870 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44342626-7e60-461f-9b1d-d99c42b41415 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:19.106146 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44342626-7e60-461f-9b1d-d99c42b41415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:19.127449 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-44342626-7e60-461f-9b1d-d99c42b41415 None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:19.128697 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-44342626-7e60-461f-9b1d-d99c42b41415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:19.195229 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1333/2654] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:47:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:19.350063 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:19.351554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:19.424745 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3f2edf-ea61-498d-8cf9-0d7b1ea035e8 service glance] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:19.430261 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:19.430898 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:19.431186 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:19.431467 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:19.441304 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:19.442943 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:19.451816 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1322/2655] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:47:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:47:29.188341 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=44 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:29.448701 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=40 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:30.725923 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51205417-6b39-4a33-bcfc-77fe567c3f99 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:30.727810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51205417-6b39-4a33-bcfc-77fe567c3f99 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:30.727810 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51205417-6b39-4a33-bcfc-77fe567c3f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:30.746594 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-51205417-6b39-4a33-bcfc-77fe567c3f99 None None] MFA Rules not processed for user `8daa1cb0ade042589bc26ecaaf59b0f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:30.747859 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-51205417-6b39-4a33-bcfc-77fe567c3f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:30.755713 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1334/2656] 10.222.0.66 () {58 vars in 958 bytes} [Fri Oct 28 20:47:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:31.020041 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:31.075046 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db13126f-51ab-46e1-9a10-8b7ec7bd0041 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:31.078670 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:31.081603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:31.081603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:31.081603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:31.082281 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:31.091780 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:31.093199 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:31.102045 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1323/2657] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:47:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:47:34.592840 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:34.593359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:34.593476 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:34.614113 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:47:34.615307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:34.622885 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1335/2658] 10.222.0.66 () {58 vars in 1009 bytes} [Fri Oct 28 20:47:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:47:34.635915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:34.648481 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf81ecd-1e53-47a2-836e-d39c06910527 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:34.653595 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:34.654275 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:34.654620 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:34.654982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:34.663921 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:34.665534 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:34.673643 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1324/2659] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:47:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:47:44.608121 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:44.670369 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=32 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:47:44.859483 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:47:44.861185 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:44.951091 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7205e34c-0b7b-4ed8-a125-7e48de296f6d None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:47:44.957194 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:47:44.957861 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:47:44.958143 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:47:44.958422 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:47:44.960259 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:47:44.973487 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:47:44.975102 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:47:44.987735 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1336/2660] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:47:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:47:54.981860 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=38 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:48:04.062344 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:48:04.072578 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d035700f-5735-4cbb-9917-f9801efdfdd9 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:48:04.076223 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:48:04.076764 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:48:04.077042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:48:04.077327 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:48:04.086258 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:48:04.087689 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:48:04.097146 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1325/2661] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:48:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 20:48:14.092841 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:48:16.058096 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:48:16.070571 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96348df0-c6d9-4d73-8acc-144b928246e5 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:48:16.076373 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd54e8c38f0f84e4287b0445947bde916', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:48:16.077064 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:48:16.077461 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:48:16.078107 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:48:16.079622 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:48:16.093380 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:48:16.095137 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:48:16.107872 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1337/2662] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:48:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:48:26.102787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=6 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:48:29.016047 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:48:29.019926 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:48:29.106075 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f63ae5b-4ede-473f-8883-eef0d5a422f0 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:48:29.111179 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:48:29.111800 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:48:29.112273 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:48:29.112554 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:48:29.114307 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:48:29.126924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:48:29.128545 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:48:29.139778 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1326/2663] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:48:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:48:29.179406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:48:29.180886 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:48:29.257150 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3c5a4a-8c7e-410e-af7b-745059d113d8 service nova] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:48:29.261887 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:48:29.262522 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:48:29.262803 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:48:29.263078 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] PATH_INFO: `/v3/users/d41ea52c88f1436188152eab37f47ab6` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:48:29.264201 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d41ea52c88f1436188152eab37f47ab6)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:48:29.266244 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:48:29.467308 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd41ea52c88f1436188152eab37f47ab6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:29.467965 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd41ea52c88f1436188152eab37f47ab6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:29.480595 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd41ea52c88f1436188152eab37f47ab6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:29.487527 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd41ea52c88f1436188152eab37f47ab6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:29.488246 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd41ea52c88f1436188152eab37f47ab6'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:29.556477 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1338/2664] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:48:29 2022] DELETE /identity/v3/users/d41ea52c88f1436188152eab37f47ab6 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:48:29.810680 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:48:29.826042 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851bc8f6-823c-41e9-a073-6d6c899eaa7f service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:48:29.882054 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:48:29.882694 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:48:29.882982 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:48:29.883265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] PATH_INFO: `/v3/projects/2d9ea1131d80461da67996b2dc844f49` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:48:29.884387 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d9ea1131d80461da67996b2dc844f49)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:48:29.886349 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:48:29.991548 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d9ea1131d80461da67996b2dc844f49', 'request_id': 'req-de05fefb-5b10-4732-9908-4f852759ffaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.059521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d9ea1131d80461da67996b2dc844f49', 'request_id': 'req-de05fefb-5b10-4732-9908-4f852759ffaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.066266 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d9ea1131d80461da67996b2dc844f49', 'request_id': 'req-de05fefb-5b10-4732-9908-4f852759ffaa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.076935 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] The token cache is being invalidate because project 2d9ea1131d80461da67996b2dc844f49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:48:30.077237 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.077910 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.079388 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1327/2665] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:48:29 2022] DELETE /identity/v3/projects/2d9ea1131d80461da67996b2dc844f49 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:48:30.083479 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:48:30.085643 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:48:30.099013 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2375daa7-f207-42a8-a7dd-c38a147e6029 admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:48:30.157927 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:48:30.158569 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:48:30.158854 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:48:30.159131 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] PATH_INFO: `/v3/users/715cfa404fcc42688a0b28db32c480d0` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:48:30.160267 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=715cfa404fcc42688a0b28db32c480d0)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:48:30.162528 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:48:30.184053 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:48:30.429336 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '715cfa404fcc42688a0b28db32c480d0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.429995 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '715cfa404fcc42688a0b28db32c480d0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.442317 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '715cfa404fcc42688a0b28db32c480d0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.449417 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '715cfa404fcc42688a0b28db32c480d0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.450076 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '715cfa404fcc42688a0b28db32c480d0'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:30.523721 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1339/2666] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:48:30 2022] DELETE /identity/v3/users/715cfa404fcc42688a0b28db32c480d0 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:48:30.770472 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:48:30.771952 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:48:30.785048 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de05fefb-5b10-4732-9908-4f852759ffaa admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:48:30.804445 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:48:30.847019 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:48:30.847594 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:48:30.847877 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:48:30.848152 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] PATH_INFO: `/v3/projects/a7b034dd1630433fa3d91432072aa33f` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:48:30.849268 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7b034dd1630433fa3d91432072aa33f)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:48:30.851300 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:48:30.973049 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7b034dd1630433fa3d91432072aa33f', 'request_id': 'req-8176c41d-3230-4659-9571-1a6020dbafee'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:31.059126 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7b034dd1630433fa3d91432072aa33f', 'request_id': 'req-8176c41d-3230-4659-9571-1a6020dbafee'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:31.066138 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7b034dd1630433fa3d91432072aa33f', 'request_id': 'req-8176c41d-3230-4659-9571-1a6020dbafee'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:31.076996 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] The token cache is being invalidate because project a7b034dd1630433fa3d91432072aa33f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:48:31.077290 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:31.077963 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:48:31.079446 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1328/2667] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:48:30 2022] DELETE /identity/v3/projects/a7b034dd1630433fa3d91432072aa33f => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 20:48:40.519232 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:DELETE=7,keystone:SELECT=20,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:48:41.073895 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:49:13.733566 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:49:13.735999 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:13.792050 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8f43fe-95d5-43af-8eb2-6bb8631bdc8b admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:49:13.842389 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5f040f-a288-44c7-9315-da59bbb5f2b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '183a4f23782143b0b234d7800b409452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:49:13.843001 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f040f-a288-44c7-9315-da59bbb5f2b2 service cinder] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:13.843280 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f040f-a288-44c7-9315-da59bbb5f2b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:13.843548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f040f-a288-44c7-9315-da59bbb5f2b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:13.845249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-9b5f040f-a288-44c7-9315-da59bbb5f2b2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:13.913353 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5f040f-a288-44c7-9315-da59bbb5f2b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:49:13.914718 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5f040f-a288-44c7-9315-da59bbb5f2b2 service cinder] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:49:13.925661 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1340/2668] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:49:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:49:16.845603 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:49:16.847165 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:16.917253 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8176c41d-3230-4659-9571-1a6020dbafee admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:49:16.920789 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91639dc527ae47e892164544572cbec4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:49:16.921439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:16.921720 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:16.921994 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:16.923690 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:16.937510 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:49:16.938915 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:49:16.949830 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1329/2669] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:49:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:49:19.356997 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] REQUEST_METHOD: `POST` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:19.357466 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:19.357598 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:19.377560 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.auth.core [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] MFA Rules not processed for user `2697ebcd1ef3492799fb7d4433b9f00f`. Rule list: `[]` (Enabled: `True`). {{(pid=81614) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 20:49:19.378755 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:19.440545 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1341/2670] 10.222.0.66 () {58 vars in 967 bytes} [Fri Oct 28 20:49:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 20:49:19.488459 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:49:19.489945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:19.559848 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8496a5-8f0c-4bad-8312-88ce7a3710c8 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:49:19.564883 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0df958d5d645473c9c0403bf9b4ee846', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:49:19.565521 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:19.565807 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:19.566124 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:19.574491 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:49:19.576061 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:49:19.584178 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1330/2671] 10.222.0.66 () {60 vars in 1384 bytes} [Fri Oct 28 20:49:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 20:49:27.634530 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:49:27.636008 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:27.641245 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=100 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:49:27.702937 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac383a5-2862-41b6-a77b-25e54ded6b66 None None] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:49:27.707492 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:49:27.708123 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:27.708406 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:27.708685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] PATH_INFO: `/v3/users/8daa1cb0ade042589bc26ecaaf59b0f7` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:27.709855 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=8daa1cb0ade042589bc26ecaaf59b0f7)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:49:27.712458 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:49:27.928884 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8daa1cb0ade042589bc26ecaaf59b0f7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:27.929539 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8daa1cb0ade042589bc26ecaaf59b0f7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:27.941787 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8daa1cb0ade042589bc26ecaaf59b0f7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:27.949211 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8daa1cb0ade042589bc26ecaaf59b0f7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:27.949883 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8daa1cb0ade042589bc26ecaaf59b0f7'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.030072 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1342/2672] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:49:27 2022] DELETE /identity/v3/users/8daa1cb0ade042589bc26ecaaf59b0f7 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:49:28.286744 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:49:28.298501 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5a641c-fb65-422c-9104-6c5624d40356 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:49:28.365013 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:49:28.365679 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:28.365964 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:28.366245 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] PATH_INFO: `/v3/projects/7b5c9ffe3749453b8d677e173d102be1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:28.367403 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b5c9ffe3749453b8d677e173d102be1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:49:28.369297 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:49:28.392851 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:49:28.501929 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b5c9ffe3749453b8d677e173d102be1', 'request_id': 'req-2c7af57c-0076-4115-bbc0-e5b219b791e8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.571143 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b5c9ffe3749453b8d677e173d102be1', 'request_id': 'req-2c7af57c-0076-4115-bbc0-e5b219b791e8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.577638 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b5c9ffe3749453b8d677e173d102be1', 'request_id': 'req-2c7af57c-0076-4115-bbc0-e5b219b791e8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.588499 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] The token cache is being invalidate because project 7b5c9ffe3749453b8d677e173d102be1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:49:28.588786 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.589439 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.590918 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1331/2673] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:49:28 2022] DELETE /identity/v3/projects/7b5c9ffe3749453b8d677e173d102be1 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:49:28.595065 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:49:28.596489 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:28.610628 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44effb06-ee65-4682-89db-3734ff4a8dcb admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:49:28.667693 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:49:28.668398 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:28.668681 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:28.668959 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] PATH_INFO: `/v3/users/46a97178b6e34e5585b353a0bd730c23` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:28.670139 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] RBAC: Authorizing `identity:delete_user(user_id=46a97178b6e34e5585b353a0bd730c23)` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:49:28.672189 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:49:28.694233 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:49:28.887307 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46a97178b6e34e5585b353a0bd730c23'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.887903 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46a97178b6e34e5585b353a0bd730c23'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.900548 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46a97178b6e34e5585b353a0bd730c23'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.908127 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46a97178b6e34e5585b353a0bd730c23'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.908768 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.notifications [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46a97178b6e34e5585b353a0bd730c23'} {{(pid=81614) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:28.979272 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1343/2674] 10.222.0.66 () {58 vars in 1224 bytes} [Fri Oct 28 20:49:28 2022] DELETE /identity/v3/users/46a97178b6e34e5585b353a0bd730c23 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:49:29.235924 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:49:29.237563 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:49:29.252872 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7af57c-0076-4115-bbc0-e5b219b791e8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:49:29.310945 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8481404e88e44e68b2c70669768eebe2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9255f3850e094deb824d58d03e377387', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:49:29.311565 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:49:29.311860 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:49:29.312136 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] PATH_INFO: `/v3/projects/dd1881d4230046a1a465f723587e6421` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:49:29.313248 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd1881d4230046a1a465f723587e6421)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:49:29.315235 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:49:29.435671 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd1881d4230046a1a465f723587e6421', 'request_id': 'req-0065f9be-54ef-49f5-80f0-9be68af13bc5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:29.503619 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd1881d4230046a1a465f723587e6421', 'request_id': 'req-0065f9be-54ef-49f5-80f0-9be68af13bc5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:29.510887 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd1881d4230046a1a465f723587e6421', 'request_id': 'req-0065f9be-54ef-49f5-80f0-9be68af13bc5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:29.521279 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] The token cache is being invalidate because project dd1881d4230046a1a465f723587e6421 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 20:49:29.521606 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:29.522265 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0065f9be-54ef-49f5-80f0-9be68af13bc5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 20:49:29.524357 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1332/2675] 10.222.0.66 () {58 vars in 1233 bytes} [Fri Oct 28 20:49:29 2022] DELETE /identity/v3/projects/dd1881d4230046a1a465f723587e6421 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 20:49:38.974289 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=20,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:49:39.518305 dv-cinder-862365-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=66,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 20:50:10.128685 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Authenticating user token {{(pid=81614) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 20:50:10.130178 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:50:10.200585 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230dec80-c244-4e15-961f-6358087c037a admin admin] Validating token access rules against request {{(pid=81614) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 20:50:10.258750 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0288f59-55f6-425c-9191-c10f0d166701 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fbacc77af9b43f1a2c1cf15cd09e337', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2399c000b584431197e605f2c47df1b0', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81614) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 20:50:10.259359 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0288f59-55f6-425c-9191-c10f0d166701 service placement] REQUEST_METHOD: `GET` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 20:50:10.259645 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0288f59-55f6-425c-9191-c10f0d166701 service placement] SCRIPT_NAME: `/identity` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 20:50:10.260503 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0288f59-55f6-425c-9191-c10f0d166701 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81614) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 20:50:10.262249 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.fernet_utils [None req-f0288f59-55f6-425c-9191-c10f0d166701 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81614) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 20:50:10.273414 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0288f59-55f6-425c-9191-c10f0d166701 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 20:50:10.276016 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0288f59-55f6-425c-9191-c10f0d166701 service placement] RBAC: Authorization granted {{(pid=81614) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 20:50:10.284896 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: [pid: 81614|app: 0|req: 1344/2676] 10.222.0.66 () {60 vars in 1365 bytes} [Fri Oct 28 20:50:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 20:50:20.280510 dv-cinder-862365-2-iscsi devstack@keystone.service[81614]: DEBUG dbcounter [-] [81614] Writing DB stats keystone:SELECT=58 {{(pid=81614) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}